Surveying system

ABSTRACT

A system is disclosed that comprises a camera module and a control and evaluation unit. The camera module is designed to be attached to the surveying pole and comprises at least one camera for capturing images. The control and evaluation unit has stored a program with program code so as to control and execute a functionality in which a series of images of the surrounding is captured with the at least one camera; a SLAM-evaluation with a defined algorithm using the series of images is performed, wherein a reference point field is built up and poses for the captured images are determined; and, based on the determined poses, a point cloud comprising 3D-positions of points of the surrounding can be computed by forward intersection using the series of images, particularly by using dense matching algorithm.

FIELD OF THE INVENTION

The present invention pertains to a surveying system comprising a cameramodule providing images for performing a SLAM- or SfM-algorithm.

The invention describes a camera module which can be attached on a poleto a GNSS-antenna or a reflector for the measurement of points withoutthe levelling step.

Moreover, the camera module enables the measurement of points where theGNSS-signal or the line-of-sight between total station and pole isinterrupted.

Moreover, from the imaging data acquired with the camera module a pointcloud of the environment can be derived.

Moreover, rectified views or orthophotos can be generated, e. g. of theterrain or a façade.

BACKGROUND

In traditional surveying with a GNSS-pole the surveyor places the poletip onto the measuring point, levels the pole and triggers themeasurement. The levelling step takes some time and—if not carried outproperly—leads to a degraded measurement result.

Surveying with a GNSS-pole is only possible at places, where the signalsof a sufficient number of GNSS satellites can be received. When thesurveyor moves close to a building, some of the satellite signals may benot receivable anymore. Thus, at such a place a measurement is notpossible at all.

A GNSS surveying system can record absolute positions with good accuracyon a global scale, e. g. 2-4 cm. However, such a system can record onlysingle points, where the operator must position the GNSS pole verticallyon top of point to be measured. The derivation of a point cloud with aGNSS-pole is not state-of-the-art.

PRIOR ART

US 2011/0064312 A1 relates to image-based geo-referencing and disclosesa combination of GNSS measurements with image processing to provide newsolutions for positioning. Stored geo-referenced images are compared(feature-correlated) with actual images made by a GNSS receiver. This isthen used to qualify the accuracy of the GNSS measurement or complementmissing parts (e.g. height information). It is also possible the otherway round, i.e. the GNSS measurement is used to update the geo-referenceof the stored images. This can also be used to determine a localcoordinate system.

US 2011/0157359 A1 discloses aligning a virtual perspective centre of acamera with the measurement (antenna) centre of a position measurementsystem. This facilitates computations in a combined image/GNSS system.WO 2011/163454 A1 discloses a method and apparatus for image-basedpositioning, tracking image features from one image to the next in orderto determine the position change of a GNSS receiver using SLAMtechniques. WO 2010/080950 A1 discloses determining orientation of aGNSS receiver from image data.

A processing of data recorded by a system with cameras requires highcomputational resources. The state-of-the-art solution is known asprocessing of data on a powerful laptop, PC or on an external cloudserver. The processing time might be quite time consuming and usually isperformed in the office.

However, for some tasks (e.g. preview of 3D image-based reconstruction)powerful in-field computational capacity is needed. Data transfer viawireless networks is usually also time consuming when the bandwidth islimited and does not allow getting quick results of computations.

The following solution particularly is proposed to have an ability of afast in-field data processing. One or more external portablecomputational devices (e.g. smartphone, tablet PC, laptop) areregistered as computational devices in the surveying system. The systemhas a cable or wireless connection with at least one of thesecomputational devices. The data are transferred to these devices and allcomputations are automatically distributed between all availablecomputational devices. All computational devices could communicatebetween each other.

Among others, one advantage of such solution is an ability to use allavailable computational resources for fast in-field data processing orvisualization. New devices could be easily added for computation withoutupdating of the surveying device.

One aspect of the invention relates to a surveying system adapted todetermine positions of a position measuring resource being mounted on asurveying pole, particularly a GNSS-antenna or a retro-reflector, in acoordinate system of the surveying system, the surveying systemcomprising a surveying subsystem with a camera module (30) beingattached to the surveying pole (10) and comprising at least one camera(31) for capturing images, and a control and evaluation unit (12) havingstored a program with program code so as to control and execute anorientation determining functionality in which

-   -   a series of images of the surrounding is captured with the at        least one camera when moving along a path through a surrounding,        the series comprising an amount of images captured with        different poses of the camera, the poses representing respective        positions and orientations of the camera;    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points,

SUMMARY

-   -   a reference point field is built up comprising a plurality of        reference points of the surrounding, wherein coordinates of the        reference points are derived, and    -   the poses for the images are determined;    -   determined positions that are adopted by the position measuring        resource when moving along the path are retrieved from the        surveying system; and    -   an external orientation of the at least one camera in the        coordinate system of the surveying system is derived at least        based on the determined pose for at least one designated image        of the series of images and on the determined positions.

The surveying system further comprises a processing unit having stored aprogram with program code so as to control and execute a single pointmeasurement functionality in which—upon a trigger—

-   -   a trigger-related position, which has been adopted by the        position measuring resource at a trigger-related point of time        when moving along the path, is determined by the surveying        system, with the use of the position measuring resource and        particularly also the determined poses and/or IMU-data;    -   a trigger-related external orientation, which has been adopted        by the at least one camera at the trigger-related point of time,        is determined by the orientation determining functionality; and    -   a position of a bottom of the pole of the surveying system is        determined in coordinates of the coordinate system based on at        least the trigger-related position and the trigger-related        external orientation.

In one embodiment of this system, the orientation of the at least onecamera is derived based on one or more of the determined poses and

-   -   based on data from an inertial measuring unit of the camera        module, and/or    -   based on a multitude of determined positions of the position        measuring resource, particular a travelling history for the        moved path.

In another embodiment of this system, the orientation is derived in allthree rotational degrees of freedom, in particular wherein position andorientation in six degrees of freedom are determined.

In another embodiment of this system, an orientation of the surveyingpole is derived based on the poses.

In a further embodiment of this system, the single point measurementfunctionality involves that a trigger-related section-wisebundle-adjustment is performed for the determination of thetrigger-related external orientation, wherein—using only a subset ofmost recent images of the series of images, the subset particularlyconsisting of between the most actual 50 and the most actual 5images—the reference point field and the poses within the range of thesubset are retroactively re-calculated for the images of the subset andwherein these re-calculated poses are used for deriving the externalorientation.

The invention also relates to a surveying subsystem providing positionalinformation of a surrounding in form of a scaled point cloud. Theinvention thus relates to a surveying subsystem comprising a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a hand-carried surveying pole,particularly wherein the position measuring resource comprises aGNSS-antenna or a retro-reflector.

The camera module is designed to be attached to the surveying pole andcomprises at least one camera for capturing images. The control andevaluation unit has stored a program with program code so as to controland execute a spatial representation generation functionality inwhich—when moving along a path through a surrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera, the series comprising an amount of images        captured with different poses of the camera, the poses        representing respective positions and orientations of the        camera,    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined,    -   based on the determined poses, a point cloud comprising        3D-positions of points of the surrounding is computed by forward        intersection using the images of the series of images,    -   determined positions of the position measuring resource for        points that have been adopted on the path are received by the        control and evaluation unit from the surveying system, and    -   the point cloud is scaled, and particularly geo-referenced, with        help of the received determined positions.

According to one specific embodiment of that invention, the control andevaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way, thatthe point cloud is generated

-   -   spatially inclusive and comprehensive across the whole        surrounding and/or    -   with comparatively low resolution of the 3d-information across        the surrounding, thus providing comparatively fast processing of        the point cloud.

According to another specific embodiment of that invention, the controland evaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way, thata graphical reproduction is generated for the scaled point cloud, thegraphical reproduction being displayable by display means of thesurveying system, thus providing for a direct feedback to a user aboutalready acquired data, so that the already acquired data can be checkedregarding its completeness.

According to yet another specific embodiment of the subsystem, thecontrol and evaluation unit is configured so that the spatialrepresentation generation functionality is controlled and executed insuch a way, that position information is derived for a single pointselected in at least one image of the series of images, wherein a subsetof images with determined poses related to the selected point isautomatically identified from the series of images, particularly allimages in which the selected point appears, and the position informationis calculated based on the subset, particularly after the point wasmanually selected by a user.

According to a further specific embodiment of the subsystem, the controland evaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way, thatthe point cloud is processed covering

-   -   the surrounding as a whole as commonly appearing at least in        pairs of images of series of images, thus providing for a global        representation with comparatively low point-to-point resolution,        and/or    -   a defined region of the surrounding as commonly appearing at        least in pairs of images of series of images, thus providing for        a regional representation with higher point-to-point resolution        compared to the global representation, particularly wherein the        point-to-point resolution of the point cloud is automatically        adapted depending on the size of the region so that processing        time fulfils a defined threshold.

Now referring to the issue of creating orthophotos, a correspondingarrangement according to that invention is described as set forth below.

The invention also relates to a surveying subsystem comprising a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a surveying pole, particularly of aGNSS-antenna or of a retro-reflector, the camera module being designedto be attached to the surveying pole and comprising at least one camerafor capturing images, the control and evaluation unit having stored aprogram with program code so as to control and execute a georeferencingfunctionality in which—when moving along a path through a surrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera, a surrounding ground appearing in the images,        the series comprising an amount of images captured with        different poses of the camera, the poses representing respective        positions and orientations of the camera;    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined;    -   based on the determined poses, a point cloud comprising        3D-positions of points of the surrounding is computed by forward        intersection using the series of images, particularly by using        dense matching algorithm; and    -   an orthorectified orthophoto of the surrounding ground is        generated based on the series of images.

In one embodiment of this subsystem, the georeferencing functionalitycomprises matching the orthophoto with a reference orthophoto forderiving position information of the surveying subsystem, in particularwherein the reference orthophoto

-   -   is stored in the control and evaluation unit, and/or    -   is an aerial image or a part thereof, in particular wherein the        aerial image is orthorectified.

In another embodiment of this subsystem, the orthorectified orthophotois generated by means of image stitching of at least two images of theseries of images.

According to another aspect regarding specific optics of a camera of acamera module, the invention also relates to a camera module to be usedas part of a surveying system that is adapted to determine positions ofa position measuring resource being mounted on a surveying pole,particularly wherein the position measuring resource comprises aGNSS-antenna or a retro-reflector. The camera module is designed to beattached to the surveying pole and comprises two cameras arrangedrelative to each other with substantially diametrically opposing viewingdirections, each of the two cameras having a fisheye lens and beingadapted to capture wide panoramic images.

The invention also relates to a surveying subsystem comprising a cameramodule and a control and evaluation unit. The camera module comprises atleast one camera, the camera having a fisheye lens and being adapted tocapture wide panoramic images. The control and evaluation unit hasstored a program with program code so as to control and execute aspatial representation generation functionality in which—when movingalong a path through a surrounding—a series of panoramic images of thesurrounding is captured with the at least one camera, the seriescomprising an amount of panoramic images captured with different posesof the camera, the poses representing respective positions andorientations of the camera, a SLAM-evaluation with a defined algorithmusing the series of panoramic images is performed, wherein a pluralityof respectively corresponding image points are identified in each ofseveral sub-groups of panoramic images of the series of panoramic imagesand, based on resection and forward intersection using the plurality ofrespectively corresponding image points, a reference point field isbuilt up comprising a plurality of reference points of the surrounding,wherein coordinates of the reference points are derived, and the posesfor the panoramic images are determined.

By use of a fisheye optical system in connection with the camera modulea comparatively wide field of view is realised providing images whichcover a correspondingly large region of a surrounding. For covering adefined part of the surrounding, fewer images have to be taken.

According to a specific embodiment of this invention, the cameras modulecomprises two cameras arranged relative to each other with substantiallydiametrically opposing viewing directions each of the two cameras havinga fisheye lens and being adapted to capture wide panoramic images.

According to another specific embodiment of that surveying subsystem,the control and evaluation unit is configured so that the spatialrepresentation generation functionality is controlled and executed insuch a way, that the poses for the panoramic images are determined byapplying a defined optical imaging method taking account of the opticalproperties of the fisheye lens, in particular wherein different imagingplanes are used with the optical imaging method and/or one of thefollowing mapping functions is used:

-   -   stereographic mapping function,    -   equidistant mapping function,    -   equisolid angle mapping function or    -   orthographic mapping function.

Calculation of the poses and/or a point cloud using images captured withsuch fisheye optics are performed using one of above methods in order toprovide an exact projection of the reference points and thus to providecorrect position and/or orientation information for the surrounding.With projecting the surrounding like that a distortion-free image of thesurrounding can be provided.

According to another specific embodiment of the subsystem, each of thetwo cameras has a field of view of at least 180° in the respectiveviewing directions regarding at least one axis, particularly in anazimuthal direction.

Using the two cameras with fisheye lenses with at least 180° concerningtheir fields of view, fully panoramic images with 360° viewing field canbe captured and thus providing for a more complete and/or more detailedgathering of data. Accordingly, more information regarding thesurrounding can be gathered, i.e. particularly, lager point clouds.

With such combination of fisheye optics with a kind of SLAM-algorithm asdescribed (i.e. capturing a series of images, identifying image pointwhich correspond to reference points in the surrounding and determiningthe poses for the images) a very efficient way of deriving a largeamount of position (e.g. for single points in the surrounding) and/ororientation information (e.g. of the panoramic camera or a pole whichthe camera is attached to) is given. A wide region of the surroundingcan be captured with the camera(s) and terrain information correspondingto the captured region can be derived based on respective panoramicimages.

Another aspect of the invention focuses on a specific camera module tobe used as part of a surveying system that is adapted to determinepositions of a position measuring resource being mounted on a surveyingpole, particularly wherein the position measuring resource comprises aGNSS-antenna or a retro-reflector, wherein the camera module is designedto be attached to the surveying pole and comprises a camera and anoptical element, the optical element being designed so that the camerahas—in an azimuthal direction—an angle of view of 360°, particularlywherein the optical element comprises a mirroring surface in form of acone, a sphere or a curvature.

The optical element preferably may be embodied as a parabolic mirror.

According to an embodiment of that camera module, the camera and theoptical element are arranged and designed so that a surrounding to becaptured is projected onto a sensor of the camera so that an image ofthe surrounding is providable with identically sized pieces of the imagebasically covering identically sized parts of the surrounding, thusproviding for an image representing the captured surrounding inhomogenous reproduction.

With projecting the surrounding in above manner onto the sensor adistortion-free image of the surrounding is provided and thus, fasterprocessing of captures images—particularly with view to SLAM processingor processing of point cloud—is enabled.

Alternatively, according to another aspect of this invention, apolarization filter particularly for reducing reflections from windowsor filtering sun-light can be mounted with the camera, e.g. in front ofthe lens of the camera.

The arrangement of such optical element (e.g. parabolic mirror) incombination with a camera as described provides for benefits to also becomparable with these regarding a combination of fisheye optics with acamera or at least two cameras. Additionally, a 360° panoramic view andcapturing of corresponding images is realised with a single camera only,thus also providing for the use of only one camera.

Now referring to the issue of triggering the cameras arranged in acamera module in order to provide synchronously captured images,corresponding arrangements according to that invention are described asset forth below.

The invention also relates to a surveying subsystem comprising a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a surveying pole, particularly of aGNSS-antenna or of a retro-reflector. The camera module is designed tobe attached to the surveying pole and comprising at least two camerasfor capturing images. The control and evaluation unit has stored aprogram with program code so as to control and execute a panoramic imagegeneration functionality in which—in response to a starting command—

-   -   a series of images of the surrounding is captured with the at        least two cameras, the series comprising an amount of images        captured with different poses of the cameras, the poses        representing respective positions and orientations of the        cameras; and    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined;

wherein a central optical trigger signal is generated that isperceivable by at least a first and a second camera and that causes thefirst and second camera to synchronously capture an image.

In one embodiment of this subsystem, the panoramic image generationfunctionality comprises generating a combined stitched image from thesynchronously captured images of the first and second camera.

In one embodiment of this subsystem, the position measuring resourcecomprises a GNSS-antenna, and the control and evaluation unit isconfigured so that—when moving along a path through the surrounding—thepanoramic image generation functionality is controlled and executed insuch a way, that

-   -   GNSS-data based on GNSS-signals received through the        GNSS-antenna is received by the control and evaluation unit from        the surveying system, and that    -   the central optical trigger signal is generated based on the        received GNSS-data.

In another embodiment of this subsystem, the control and evaluation unitis configured so that—when moving along a path through thesurrounding—the panoramic image generation functionality is controlledand executed in such a way, that the starting command is released basedon

-   -   a defined positional grid regarding a density of panoramic        images to be generated and    -   a travelling history for the moved path, the travelling history        being derived from the received determined positions.

In a further embodiment the subsystem comprises at least one opticaltriggering means that is perceivable by each of the at least twocameras, wherein the central optical trigger is a perceivable opticalsignal of the at least one optical triggering means, in particularwherein the optical triggering means comprises a flash light.

In a another embodiment the subsystem comprises at least one opticaltriggering means that is perceivable by the first and second camera,wherein the central optical trigger is a perceivable optical signal ofthe at least one optical triggering means, in particular wherein theoptical triggering means comprises a light emitting diode.

The invention also relates to a surveying subsystem comprising a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a surveying pole, particularly of aGNSS-antenna or of a retro-reflector. The camera module is designed tobe attached to the surveying pole and comprises at least one camera forcapturing images.

The control and evaluation unit has stored a program with program codeso as to control and execute an image triggering functionality inwhich—when moving along a path through a surrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera, the series comprising an amount of images        captured with different poses of the camera, the poses        representing respective positions and orientations of the        camera; and    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined;

wherein the surveying subsystem comprises means for determining at leastone pre-defined external trigger signal, and the image triggeringfunctionality comprises determining an external trigger signal, whereinbased on the determined external trigger signal the at least one cameracaptures an image.

In one embodiment of this subsystem, the camera module comprises atleast two cameras; the external trigger signal causes each of the atleast two cameras to synchronously capture an image; and a combinedstitched image is generated from the synchronously captured images ofthe at least two cameras.

In another embodiment, the subsystem comprises means for storing atleast one pre-defined external trigger signal, wherein the at least onepre-defined external trigger signal comprises

-   -   at least one pre-defined gesture of the user being determinable        by the at least one camera;    -   at least one acoustic signal; and/or    -   a contacting of the surveying pole, particularly a bottom end of        the surveying pole, with the ground.

The invention also relates to a blurring functionality of of a surveyingsystem. On the image data, algorithms for face detection and detectionof license plates on cars can be applied. In order to protect theprivacy of people, the detected faces or license plates can then be madeunrecognizable, e. g. by blurring the corresponding areas in the imagesand also in the point cloud.

For image data recorded with a frame rate of about 10 FPS and morefeature tracking algorithms can be applied, e. g. Kanade-Lucas-Tomasi(KLT). Alternatively, particularly for still images the detection ofcorresponding features algorithms such as SIFT, SURF, BRISK, BRIEF, etc.can be applied.

A surveying subsystem according to this aspect of the inventioncomprises a camera module and a control and evaluation unit to be usedas part of a surveying system that is adapted to determine positions ofa position measuring resource being mounted on a surveying pole,particularly of a GNSS-antenna or of a retro-reflector. The cameramodule is designed to be attached to the surveying pole and comprises atleast one camera for capturing images. The control and evaluation unithas stored a program with program code so as to control and execute adata reduction functionality in which—when moving along a path through asurrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera, the series comprising an amount of images        captured with different poses of the camera, the poses        representing respective positions and orientations of the        camera;    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined;    -   based on the determined poses, a point cloud comprising        3D-positions of points of the surrounding is computed by forward        intersection using the series of images, particularly by using        dense matching algorithm;    -   objects of a pre-defined and/or user-defined kind are identified        in the images; and    -   3D-positions of points being connected to the identified objects        in the image are modified or deleted.

In one embodiment of this subsystem, the 3D-positions are modified ordeleted in such a way that individual characteristics of the identifiedobjects are not identifiable in the computed point cloud.

In another embodiment of this subsystem, image data being connected tothe identified objects in the image are modified or deleted,particularly in such a way that individual characteristics of theidentified objects are not identifiable in the image.

In a further embodiment of this subsystem, pre-defined kinds of objectscomprise at least

-   -   a user operating the surveying system;    -   parts of the surveying system;    -   human faces; and/or    -   vehicle license plates.

Another aspect of the invention focuses on privacy or secrecy protectionwhen performing surveying tasks. In certain areas, taking pictures orvideos is subject to approval of the owners or authorities or completelyprohibited. Such areas are e. g. military facilities. According to thisaspect of the invention, in order to avoid legal conflicts in the courseof a surveying task, all images taken for the computing of a pointcloud, are deleted directly after they have been used accordingly andare not needed any longer for computing the point cloud. Only a smallnumber of images have to be stored at the same time for computing thepoint cloud.

Therefore, the invention also relates to a surveying subsystemcomprising a camera module and a control and evaluation unit to be usedas part of a surveying system that is adapted to determine positions ofa position measuring resource being mounted on a surveying pole,particularly of a GNSS-antenna or of a retro-reflector. The cameramodule is designed to be attached to the surveying pole and comprises atleast one camera for capturing images. The control and evaluation unithas stored a program with program code so as to control and execute alocalization and/or mapping functionality in which—when moving along apath through a surrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera, a surrounding ground appearing in the images,        the series comprising an amount of images captured with        different poses of the camera, the poses representing respective        positions and orientations of the camera;    -   the images are stored in a memory of the control and evaluation        unit, particularly in a volatile memory;    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined;    -   based on the determined poses, a point cloud comprising        3D-positions of points of the surrounding is computed by forward        intersection using the series of images, particularly by using        dense matching algorithm; and    -   each of the images is deleted after having been used for        computing the point cloud.

In one embodiment of this subsystem, the images are deleted withoutdelay after having been used for computing the point cloud, inparticular wherein the images are deleted in such a way that no videostream can be created from the deleted images.

In another embodiment of this subsystem, the images are stored anddeleted in such a way that not more than ten images are stored in thememory at the same time, in particular not more than five images.

Now referring to an inventive aspect of fading out objects from gatheredimages in order to improve processing, according to that invention, asurveying subsystem comprises a camera module and a control andevaluation unit to be used as part of a surveying system that is adaptedto determine positions of a position measuring resource being mounted ona hand-carried surveying pole, particularly wherein the positionmeasuring resource comprises a GNSS-antenna or a retro-reflector. Thecamera module is designed to be attached to the surveying pole andcomprises at least one camera for capturing images.

The control and evaluation unit has stored a program with program codeso as to control and execute a spatial representation generationfunctionality in which—when moving along a path through a surrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera, the series comprising an amount of images        captured with different poses of the camera, the poses        representing respective positions and orientations of the        camera,    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined,

wherein in the series of images an interfering object is recognised byimage processing, feature recognition techniques and/or comparison ofthe images, and the recognised interfering object is faded out inconcerned images regarding identifying the set of image points and/ordetermining the poses for the images, so that evaluation is lesseffortful and interference suppressed.

According to an embodiment of that surveying subsystem, the control andevaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way, thata moving object is recognised as the interfering object on basis of amotion detection algorithm, particularly by feature tracking.

According to another specific embodiment of the subsystem, a person,particularly the user carrying the pole, or a car is recognised as theinterfering object.

According to a further embodiment of that invention, the control andevaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way thatapplying the steps of recognising the interfering object and fading outthe interfering object is triggered by a user command.

In particular, the user coarsely marks an object to be recognised asinterfering object in at least one of the series of images, whereforethe at least one image is displayed on displays means.

The detection and fading out of interfering objects provides for afaster data processing as less amount of image data has to be consideredfor determining poses. Furthermore, the calculated poses comprise ahigher degree of precision as incorrect imaging—which would occur e.g.if considering a moving car in more images which were taken at differentpoints in time—initially is suppressed.

Additionally, above method generally provides for better (particularlymore precise and faster) matching and identification of reference points(image points in the images). The method may be executed automaticallycontrolled by the control and evaluation unit or may be triggered byuser, e.g. when the user marks a respective object in an image displayedon display means of the surveying system.

Another aspect focuses on calibration of cameras (a camera modulerespectively).

According to that invention, a surveying subsystem comprises a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a surveying pole, particularlywherein the position measuring resource comprises a GNSS-antenna or aretro-reflector. The camera module is designed to be attached to thesurveying pole and comprises at least one camera for capturing images.The position measuring resource comprises a reference pattern located atthe housing of the measuring resource. The position measuring resourceand the camera module are arranged in defined manner relative to eachother so that the reference pattern appears in the field of view of theat least one camera.

Furthermore, the control and evaluation unit has stored a program withprogram code so as to execute a camera calibration functionality inwhich

-   -   an image covering the reference pattern is captured by the at        least one camera, and    -   calibration parameters regarding a fixed spatial relationship        between the position measuring resource and the at least one        camera are determined on basis of the captured image and a        pre-known reference image, the reference image representing a        defined position and orientation of the at least one camera        relative to the position measuring resource by a defined        appearance of the reference pattern in the reference image.

According to a specific embodiment of that surveying subsystem, thereference pattern is embodied as a structured pattern being provided bya light source or as a permanent pattern.

Based on the determined calibration data, a verification of a desiredposition of the camera or the camera module, respectively, relative tothe position measuring resource is enabled. If the reference patternappears in a captured image according to a target appearance, the camerais positioned in an expected home position. In case the appearance ofthe pattern differs from an expected appearance the camera module isposition and/or oriented different to a home position. A deviation ofthe position of the camera can be derived by comparing the referenceimage with the actually captured image, particularly by comparing theappearances of the patterns.

The invention also relates to a surveying subsystem comprising a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a hand-carried surveying pole,particularly wherein the position measuring resource comprises aGNSS-antenna or a retro-reflector. The camera module is designed to beattached to the surveying pole and comprises at least one camera forcapturing images. The position measuring resource and the camera moduleare arranged in defined manner relative to each other. Furthermore, thecontrol and evaluation unit has stored a program with program code so asto control and execute a calibration functionality in which—when movingalong a path through a surrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera, the series comprising an amount of images        captured with different poses of the camera, the poses        representing respective positions and orientations of the        camera,    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined,    -   determined positions of the position measuring resource for        points that have been adopted on the path are received by the        control and evaluation unit from the surveying system, and    -   calibration parameters regarding a fixed spatial relationship        between the position measuring resource and the at least one        camera are derived based on an interrelated assessment of the        received determined positions and the determined poses.

According to that subsystem, determined positions of the positionmeasuring resource for points that have been adopted on the path arereceived by the control and evaluation unit from the surveying system,and calibration parameters regarding a fixed spatial relationshipbetween the position measuring resource and the at least one camera arederived based on an interrelated assessment of the received determinedpositions and the determined poses.

According to an embodiment of the invention, the control and evaluationunit is configured so that the calibration functionality is controlledand executed in such a way, that trajectories according to the path arederived based on the derived poses and on the received determinedpositions, wherein the calibration parameters are derived based oncomparing the trajectories.

According to another specific embodiment of the invention, the controland evaluation unit is configured so that the calibration functionalityis controlled and executed in such a way that the calibration parametersare derived by additionally using data generated by an inertialmeasuring unit, the inertial measuring unit being associated with thesurveying subsystem or the position measuring resource.

According to yet another specific embodiment of the invention, thereceived positions of the position measuring resource are determined byreceiving GNSS-signals on side of the position measuring resource or byreflecting a measuring laser beam on side of the position measuringresource, the measuring laser beam being emitted and received by a totalstation or a theodolite.

Using information regarding both the position information received fromthe surveying system and the poses determined based on the capturedimages relative positions and orientation of the camera (camera module)and the position measuring resource can be derived and a relativespatial orientation of the two components can be calculated therefrom.The poses represent respective orientations of the camera. That methodprovides for a continuous and automatic position-calibration whilemoving the camera (and the position measuring resource accordingly).With help of data provided by the inertial measuring unit suchcalibration is enable with six degrees of freedom (6-DOF).

The invention also relates to a surveying subsystem comprising a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a hand-carried surveying pole,particularly wherein the position measuring resource comprises aGNSS-antenna or a retro-reflector. The camera module is designed to beattached to the surveying pole and comprises at least one camera forcapturing images. An inertial measuring unit is associated with thesurveying subsystem or the position measuring resource in a fixedspatial relationship relative to the surveying subsystem.

The control and evaluation unit has stored a program with program codeso as to control and execute a calibration functionality in which—whenmoving along a path through a surrounding—

-   -   inertial measuring data is gathered while moving along the path,    -   camera-based localisation data is generated, wherefore a series        of images of the surrounding is captured with the at least one        camera, the series comprising an amount of images captured with        different poses of the camera, the poses representing respective        positions and orientations of the camera, a SLAM-evaluation with        a defined algorithm using the series of images is performed,        wherein a plurality of respectively corresponding image points        are identified in each of several sub-groups of images of the        series of images and, based on resection and forward        intersection using the plurality of respectively corresponding        image points, a reference point field is built up comprising a        plurality of reference points of the surrounding, wherein        coordinates of the reference points are derived, and the poses        for the images are determined, and    -   calibration parameters for the inertial measuring unit are        derived based on the gathered inertial measuring data and the        camera-based localisation data, particularly wherein a Kalman        filter is used.

According to an embodiment of the invention, the control and evaluationunit is configured so that the calibration functionality is controlledand executed in such a way, that the inertial measuring unit iscalibrated using the derived calibration parameters, wherein asystematic error of the inertial measuring unit is compensated,particularly wherein a bias of the inertial measuring unit iscompensated.

By applying the derived calibration parameters a calibration of the IMU(inertial measuring unit) is provided just (only) on basis of theorientation data derived from the poses of the images, i.e. derived froma series of images captured while moving the camera. Camera and IMU arearranged in defined relative spatial positions while moving.

The invention also relates to a surveying subsystem comprising a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions with use of asurveying pole, The camera module is designed to be attached to thesurveying pole in a known distance to a bottom end of the pole andcomprises at least one camera for capturing images. The control andevaluation unit has stored a program with program code so as to controland execute a scaling functionality in which—when moving along a paththrough a surrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera, a surrounding ground appearing in the images,        the series comprising an amount of images captured with        different poses of the camera, the poses representing respective        positions and orientations of the camera;    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined;    -   based on the determined poses, a point cloud comprising        3D-positions of points of the surrounding is computed by forward        intersection using the series of images, particularly by using        dense matching algorithm;    -   a distance from the camera to the ground is determined based on        the known distance from the camera to the bottom end; and    -   the point cloud is scaled based on the determined distance.

In one embodiment of this subsystem, scaling the point cloud comprisesdetermining 3D-positions of points on the ground based on the determineddistance.

In another embodiment of this subsystem, determining the distance fromthe camera to the ground comprises deriving an orientation of thecamera, in particular from the poses.

Furthermore, another aspect concerning the invention relates to abundle-adjustment for a surveying subsystem, the surveying subsystemcomprising a camera module and a control and evaluation unit to be usedas part of a surveying system that is adapted to determine positions ofa position measuring resource being mounted on a hand-carried surveyingpole, particularly wherein the position measuring resource comprises aGNSS-antenna or a retro-reflector.

The camera module is designed to be attached to the surveying pole andcomprises at least one camera for capturing images. The control andevaluation unit has stored a program with program code so as to controland execute a spatial representation generation functionality inwhich—when moving along a path through a surrounding—at least

-   -   a series of images of the surrounding is captured with the at        least one camera, the series comprising an amount of images        captured with different poses of the camera, the poses        representing respective positions and orientations of the        camera,    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined,    -   determined positions of the position measuring resource for        points that have been adopted on the path are received by the        control and evaluation unit from the surveying system, and,    -   upon receiving a measurement-trigger, a section-wise        bundle-adjustment is performed, in which—using only a subset of        the series of images with a reduced number of images being        related to the measurement-trigger—the reference point field and        the poses are retroactively re-calculated for the images of the        subset.

According to the invention, based on the determined poses, a point cloudcomprising 3D-positions of points of the surrounding is computed byforward intersection using the images of the series of images,determined positions of the position measuring resource for points thathave been adopted on the path are received by the control and evaluationunit from the surveying system, and on selection of a single point or aregion in at least one of the images of the series of images,bundle-adjustment for refining the point cloud is performed based atleast on image data of a pre-defined area around the point or of theregion respectively and on information relating to the receiveddetermined positions.

Particularly, as a final or intermediate step, the overall solution (atleast poses and point cloud) is refined using bundle-adjustment. Thispart of the algorithm is a non-linear least squares minimization of there-projection error. It will optimise the location and orientation ofrespective camera positions and 3D-points.

In context of above bundle-adjustment, a part of gathered measuring data(e.g. images, image points, poses and/or 3D-positions) is used forproviding a refined (partial) point cloud referring to a part of thesurrounding covered by the captured images. In particular, suchmeasuring data provided by a Kalman-Filter is used. Particularly,measuring data according to a close measuring history is used, e.g. datacorresponding to the preceding and/or successive ten poses (with respectto that image which the point or region of interest forbundle-adjustment is selected in).

According to one embodiment of the subsystem, positional information isderivable from the measurement-trigger, and the subset is selected basedon the positional information.

According to another embodiment of the subsystem, themeasurement-trigger is a trigger receivable from surveying system.

According to yet another embodiment of the subsystem, themeasurement-trigger is automatically caused by putting the pole on ameasuring point and/or keeping the pole basically in fixed position andorientation for a pre-determined time period.

In a further embodiment of the subsystem, the section-wisebundle-adjustment is performed by using non-linear least squaresminimisation of errors occurring with identifying the image points anddetermining the poses.

In one embodiment of the subsystem, localization information related tothe measurement-trigger is derived based on re-calculated poses, inparticular wherein a position and orientation of the hand-carriedsurveying pole for one or more measurement-trigger-related points oftime are derived.

In another embodiment of the subsystem, the point cloud is scaled withhelp of the received determined positions, and/or coordinates of aposition of a tip of the surveying pole is derived and output inreaction upon the measurement-trigger after performance of thesection-wise bundle-adjustment.

Performing the bundle-adjustment provides very precise determination ofa position of the selected point or, respectively, precise determinationof an amount of points in a defined region and thus providing for anexact digital 3D-model of the defined region.

By bundle-adjusting a refinement of the calculated poses regardingimages which are taken into account for execution of thebundle-adjustment is realised. Such refinement leads to more precisedetermination of 3D-positions again.

Furthermore, an orientation of the pole—if calculated from the poses—maybe determined in more precise manner. Particularly, such orientation isderived with six degrees of freedom (6-DOF) due to the computed poses.

The bundle-adjustment particularly is performed when a user of thesystem selects the point or the region, wherein the adjusted data isprovided to the user in real-time manner, i.e. the user selects thepoint and basically immediately receives the calculation results e.g. ondisplay means of the system.

A further inventive aspect relates to a specific camera module to beused as part of a surveying system that is adapted to determinepositions of a position measuring resource being mounted on a surveyingpole, particularly wherein the position measuring resource comprises aGNSS-antenna or a retro-reflector. That camera module is designed to beattached to the surveying pole and comprises at least a first and asecond camera, the first camera comprising first imaging properties andthe second camera comprising second imaging properties different fromthe first imaging properties.

According to an embodiment of the invention, the first camera isdesigned so that images with an image resolutions lower than an imageresolution of images capturable with the second camera are captured.

According to a further embodiment of the invention, the first cameracomprises a first imaging sensor and the second camera comprises asecond imaging sensor, wherein the first imaging sensor provides a lowerpoint-to-point resolution regarding images to be captured compared withthe second imaging sensor.

According to a further embodiment of the invention, the first camera isdesigned for capturing images with a higher frame rate compared to thesecond camera.

According to a further embodiment of the invention, the first camera isdesigned so that image capturing according to a first spectral range isprovided, and the second camera is designed so that image capturingaccording to a second spectral range is provided, wherein the firstspectral range differs from the second spectral range, in particularwherein the first camera provides image capturing according to aninfrared spectral range or provides thermographical images.

Capturing images with different spectral ranges provides for gettingmore detailed information of the covered surrounding, particularly forgeneration of images (e.g. by merging images of different spectralranges covering common areas of the surrounding) with improved texturesor extended information regarding spectral properties, e.g. regarding athermal property of an object.

The invention also related to a surveying subsystem comprising a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a hand-carried surveying pole,particularly wherein the position measuring resource comprises aGNSS-antenna or a retro-reflector.

The camera module is designed to be attached to the surveying pole andcomprises at least two cameras for capturing images, wherein a first ofthe at least two cameras is designed to capture first images with alower image resolutions compared to second images capturable with asecond of the at least two cameras.

The control and evaluation unit has stored a program with program codeso as to control and execute a spatial representation generationfunctionality in which—when moving along a path through a surrounding—

-   -   a first series of images of the surrounding is captured with the        first camera, the first series comprising an amount of images        captured with different poses of the first camera, the poses        representing respective positions and orientations of the first        camera, and a second series of images of the surrounding is        captured in parallel with the second camera, and    -   a SLAM-evaluation with a defined algorithm using only the first        series of images is performed, wherein a plurality of        respectively corresponding image points are identified in each        of several sub-groups of images of the first series of images        and, based on resection and forward intersection using the        plurality of respectively corresponding image points, a        reference point field is built up comprising a plurality of        reference points of the surrounding, wherein coordinates of the        reference points are derived, and the poses for the images are        determined,

thus providing for a faster data processing within the SLAM-evaluationcompared to processing data generated by capturing images with thesecond camera.

According to an embodiment of the invention, the control and evaluationunit is configured so that the spatial representation generationfunctionality is controlled and executed in such a way that based on thedetermined poses a point cloud comprising 3D-positions of points of thesurrounding is computed as the spatial representation by forwardintersection using images of the second series of images.

According to another embodiment of the invention, processing of thepoint cloud is triggered by a user command.

According to another embodiment of the invention, the control andevaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way, thatprocessing of the point cloud is performed using a defined subset ofimages of the second series of images, wherein the subset is selectableby a user, thus providing for generating detailed 3D-inforamtion for adesired part of the surrounding in comparatively short time.

According to another embodiment of the invention, the control andevaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way, thata dense matching algorithm is executed for providing the point cloudusing images of the second series of images.

According to another embodiment of the invention, the control andevaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way, thatbased on the determined poses, an initial point cloud comprising3D-positions of points of the surrounding is computed by forwardintersection using the images of the first series of images only.

According to a further specific embodiment of the invention, the controland evaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way, thata graphical reproduction is generated for the initial point cloud, thegraphical reproduction being displayable by display means of thesurveying system, thus providing for a comparatively fast directfeedback to a user about already acquired data, so that the alreadyacquired data can be checked regarding its completeness.

According to a further specific embodiment of the invention, the controland evaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way, thata dense matching algorithm is executed for providing the initial pointcloud using the images of the first series of images with lower imageresolution, thus providing comparatively fast data processing.

According to a further specific embodiment of the invention, the controland evaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way, thatthe first camera captures the first images with a higher frame ratecompared to capturing the second images by the second camera.

Consequently, as both, images with high resolution and images with alower resolution, are provided by the at least two cameras of the cameramodule, the aspect of a fast and efficient data processing together withthe possibility of generating highly precise data concerningdetermination of positions and/or orientations is given. Thus,particularly depending on the type or stage of a measuring process auser is enabled to choose which data is to be calculated and/or outpute.g. on display means. E.g. for getting a fast overview regarding theprogress of the measuring process fast data processing and output wouldbe required and would be providable by image data of lower resolution.

Alternatively or additionally, very detailed and precise positioninformation is provided by e.g. simultaneously using higher resolutionimage data.

Another aspect of the invention relates to a similar approach concerninga surveying subsystem with a specific camera. The surveying subsystemcomprises a camera module and a control and evaluation unit to be usedas part of a surveying system that is adapted to determine positions ofa position measuring resource being mounted on a hand-carried surveyingpole, particularly wherein the position measuring resource comprises aGNSS-antenna or a retro-reflector. The camera module is designed to beattached to the surveying pole and comprises at least one (dual-mode)camera for capturing images, the camera being designed so that differentcapturing modes regarding capturing images with different resolutionsand/or with different frame rates are provided. The control andevaluation unit has stored a program with program code so as to controland execute a spatial representation generation functionality in which

-   -   a low-resolution-high-frame-rate mode is provided for        high-frequency capturing of images with comparatively low        resolution with the at least one camera, and    -   a high-resolution-low-frame-rate mode is provided for        high-resolution capturing of images with comparatively low frame        rates with the at least one camera, and—when moving along a path        through a surrounding—    -   a series of images of the surrounding is captured using the        low-resolution-high-frame-rate mode, the series comprising an        amount of images captured with different poses of the camera,        the poses representing respective positions and orientations of        the camera, and    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined.

According to a specific embodiment of above invention, the control andevaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way, that

-   -   a series of images of the surrounding is captured using the        high-resolution-low-frame-rate mode, the series comprising an        amount of images captured with different poses of the camera,    -   the low-resolution-high-frame-rate mode and the        high-resolution-low-frame-rate mode are executed simultaneously        and    -   poses for images captured in high-resolution-low-frame-rate mode        are derived based on the image points and/or on the poses        determined for the images captured in        low-resolution-high-frame-rate mode.

According to another specific embodiment of the subsystem, the controland evaluation unit is configured so that the spatial representationgeneration functionality is controlled and executed in such a way, thatimage capturing in high-resolution-low-frame-rate mode is triggered by adefined time signal, particularly a time signal providing a triggersignal in a constant time interval, or is triggered by comparingrelative positions of the pole, the positions being derived from theposes determined for the images captured inlow-resolution-high-frame-rate mode, wherein capturing of a successiveimage is triggered depending on whether a pre-defined relative positionthreshold is arrived or exceeded, in particular wherein capturing of thesuccessive image is triggered if the difference between aprevious-pole-position and an actual-pole-position is at least onemeter, particularly two or five metres.

The use of such a camera which is embodied for capturing images withdifferent frame rates and/or resolutions provides for combining low- andhigh-resolution images within one common measuring process. The lowresolution images thus provide—due to the large amount if images,considered image points and calculated poses—a precise and fastcomputing of the poses for the images and a (low-resolution) pointcloud. Additionally, the high resolution images can be considered forprecisely determining positions of points captured in the images,orientations of e. g. the pole or a more precise (high-resolution) pointcloud.

For computing such a high-resolution point-cloud, poses for thehigh-resolution images can be determined based on the data derived withand for the low-resolution images, i.e. image points identified in theseimages and/or poses derived for these images are considered fordetermining the poses of the high-resolution images. Suchpose-determination is enabled based on using a common time reference forcapturing low-resolution as well as high-resolution images. Respectivehigh-resolution images thus can be assigned to respective low-resolutionimages.

Additionally, in particular, a user is enabled to manually trigger thecapturing of a high resolution image of a point or region of interestand thus to provide more detailed information of that point and/orregion.

Furthermore, another aspect concerning the invention relates toloop-closing with respect to data gathered by a surveying subsystem. Thesurveying subsystem comprises a camera module and a control andevaluation unit to be used as part of a surveying system that is adaptedto determine positions of a position measuring resource being mounted ona hand-carried surveying pole, particularly wherein the positionmeasuring resource comprises a GNSS-antenna or a retro-reflector. Thecamera module is designed to be attached to the surveying pole andcomprises at least one camera for capturing images.

The control and evaluation unit has stored a program with program codeso as to control and execute a spatial representation generationfunctionality in which—when moving along a path through a surrounding—

-   -   a first series of images of the surrounding is captured with the        at least one camera, the first series comprising an amount of        images captured with different poses of the cameras, the poses        representing respective positions and orientations of the        cameras,    -   an initial set of image points is identified based on the first        series of images, the initial image points representing        reference points of a reference point field, wherein each        reference point appears in at least two images of the series of        images, and    -   the poses for the images are determined based on resection using        the initial image points.

Furthermore, according to this aspect of the invention

-   -   a second series of images of the surrounding is captured with        the at least one camera,    -   reference points of the reference point field appearing in at        least one of the images of the second series of images are        identified,    -   a further set of image points is identified in the images of the        second series of images corresponding to the identified        reference points of the second series of images and    -   the poses for the images of the second series of images are        determined based on resection using the initial set and the        further set of image points.

According to an embodiment of above invention, the camera modulecomprises at least two cameras arranged relative to each other so thatpanoramic images with a field of view of 360° in azimuthal direction arecapturable. In particular, each camera comprises fisheye optics.

By actually identifying and considering reference points which alreadyhave been identified in a group of initial or preceding referencepoints, these initially identified point can be considered with actualpose reconstruction and/or point cloud calculation and a alreadycomputed pint cloud can be closed in order to represent the wholesurrounding by one completed cloud.

The repeatedly later use of earlier defined reference points alsoprovides for a refinement of gathered position and/or orientation data,as accumulated errors (e.g. occurring with determining successive poses)can be compensated by that.

If multiple images of the same region in the surrounding (or of parts ofit) are captured, these images and/or respective reference pointidentified in the images can be matched.

This at least will increase the overall accuracy.

A further inventive aspect is related to a position measuring resourcewith integrated camera. The position measuring resource is embodied asbeing mountable on a surveying pole to be used as part of a surveyingsystem that is adapted to determine positions of the position measuringresource. The position measuring resource comprises a GNSS-antennaand/or a retro-reflector and a camera-arrangement with at least onecamera for capturing images of a surrounding. The camera-arrangement andthe GNSS-antenna and/or the retro-reflector are integrated into a singlecommon non-divisible housing of the position measuring resource, and thecommon housing comprises a coupling unit designed so that the positionmeasuring resource is modularly attachable to the pole.

According to an embodiment of above invention, the camera-arrangement isbuilt and designed in such a way that panoramic images with a field ofview of 360° at least in azimuthal direction are capturable,particularly wherein the camera-arrangement comprises at least twocameras, especially at least four cameras, arranged in the commonhousing.

According to a further embodiment of the above invention, the positionmeasuring resource comprises a sensor unit being integrated into thecommon housing, the sensor unit comprising at least one of the followingsensors:

-   -   inertial measuring unit,    -   gyroscope,    -   tilt sensor,    -   accelerometer and/or    -   magnetic compass.

By arranging the camera inside the housing of the position measuringresource, which additionally comprises the GNSS-antenna or theretro-reflector, a compact component is provided, which provides forposition determination of the components and for determining anorientation of the component e.g. by performing a SLAM-algorithm or aSfM-algorithm using images captured by the camera.

Therefore, a continuous determination of six degrees of freedom (6-DOF)regarding the position and orientation of the component or the pole isprovided, as well.

Now referring to the issue of damping the camera arranged in a cameramodule in order to provide more stable capturing of images, acorresponding arrangement according to that invention is described asset forth below.

The invention refers to a camera module to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a surveying pole, particularlywherein the position measuring resource comprises a GNSS-antenna or aretro-reflector, the camera module being designed to be attached to thesurveying pole. The camera module is designed to be attached to thesurveying pole and comprises a housing and a camera-arrangement with atleast one camera, wherein the camera is integrated in the housing and ismovably mounted relative to the housing by use of a damping element insuch a way that, when moving with the surveying pole—having rigidlyattached the camera module—along a path, mechanical shocks effectuatedby touching down the surveying pole are compensated for on the side ofthe camera-arrangement, and/or in such a way that at least an azimuthalorientation of the camera-arrangement is variable with respect to anazimuthal orientation of the housing so that, when moving with thesurveying pole—having rigidly attached the camera module—along a path,alterations of the azimuthal orientation of the pole and the housing arecompensated for on the side of the camera-arrangement, particularlywherein higher frequency alterations having low amplitude arecompensated for, especially such that a relative azimuth angle beingformed between the azimuthal orientation of the camera-arrangement and amoving direction on the path is kept substantially constant in eachpoint on the path.

According to an embodiment of above invention, the actual movingdirection or a smoothed moving direction is used as the movingdirection, the smoothed moving direction particularly being derived foreach point on the path by smoothing a most recently travelled segment ofthe path.

According to a further embodiment of above invention, thecamera-arrangement is mounted relative to the housing by use of a gimbalmounting.

According to a further embodiment of the above invention, the cameramodule comprises a flywheel and/or a gyro unit being arranged in thehousing, wherein the flywheel or the gyro unit is connected with thecamera-arrangement such that forces applied by the rotating flywheel orthe gyro unit are transferred to the camera-arrangement, thus providinga stabilisation of the azimuthal orientation of the camera-arrangement.

According to a further embodiment of the above invention, the cameramodule comprises an actuator, particularly a piezo actuator, beingconnected to the camera-arrangement to actively stabilize the azimuthalorientation of the camera-arrangement.

According to yet a further embodiment of the above invention, the cameramodule comprises a unit for deriving an orientation of the housing,wherein orientation information generated by the unit is provided to theactuator for actively compensating the orientation of thecamera-arrangement according to the detected orientation of the housing.Particularly, the orientation information is generated based on

-   -   a SLAM-evaluation using a series of images captured with the at        least one camera, within which poses for the images are        determined,    -   determined positions of the position measuring resource received        by the unit from the surveying system, and/or    -   an inertial measurement unit providing for IMU measurement data.

The above invention provides for a damping of the camera when moving thehousing or the pole on which the camera module is attached. Such dampingenables to capture better quality images while moving through thesurrounding, i.e. stabilised images.

Furthermore, another aspect concerning the invention relates toquality-indication with respect to a point data gathered by a surveyingsubsystem, particularly a point cloud or a remote single pointmeasurement.

The respective surveying subsystem—according to that inventiveaspect—comprises a camera module and a control and evaluation unit to beused as part of a surveying system that is adapted to determinepositions of a position measuring resource being mounted on ahand-carried surveying pole, particularly wherein the position measuringresource comprises a GNSS-antenna or a retro-reflector. The cameramodule is designed to be attached to the surveying pole and comprises atleast one camera for capturing images.

The control and evaluation unit has stored a program with program codeso as to control and execute a spatial representation generationfunctionality in which—when moving along a path through a surrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera, the series comprising an amount of images        captured with different poses of the camera, the poses        representing respective positions and orientations of the        camera,    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined, and    -   a quality indicative output is generated concerning the        measurement uncertainty of a 3D-position for at least one point        of the surrounding to be computable by forward intersection        using the images of the series of images and the determined        poses.

According to an embodiment of the above aspect, the forward intersectionimplies using a variance-covariance-matrix, particularly wherein theforward intersection is carried out by use of a least square method, andthe quality indicative output is derived from thevariance-covariance-matrix.

According to another embodiment of this aspect, the quality indicativeoutput represents an uncertainty regarding the precision with which the3D-position for the at least one point is computable or computed,particularly wherein the quality indicative output is influenced atleast by

-   -   a precision with which image-positions for the image points        within the images is determinable,    -   an amount of images which are usable out of the series of images        for the computation of the 3D-position for at least one point of        the surrounding and    -   a length of the baselines of the poses of the usable images.

According to another embodiment of this aspect, based on the determinedposes, a point cloud comprising 3D-positions of points of thesurrounding is computed by forward intersection using the images of theseries of images and the quality indicative output is generated for atleast one of the computed points of the point cloud, particularlywherein the quality indicative output is generated for a subset ofpoints of the point cloud referring to a corresponding amount of poseswhich the subset is determined from.

According to a further embodiment of the inventive aspect above, agraphical reproduction is generated for the point cloud representing thequality indicative output, the graphical reproduction being displayableby display means of the surveying system, thus providing for a feedbackto a user about already acquired data, so that the already acquired datacan be checked regarding its quality.

According to a further embodiment of the inventive aspect, the qualityindicative output is represented by a coloured point cloud, a scaledindication or a specific sign displayable on display means of thesurveying system.

According to a further embodiment of the inventive aspect above,determined positions of the position measuring resource for points thathave been adopted on the path are received by the control and evaluationunit from the surveying system, and the point cloud is scaled with helpof the received determined positions.

The above-described inventive aspect provides for generating informationrepresenting a quality of the data derived on basis of captured images.Particularly, a user is provided with such information allowingre-planning or adapting a measuring process in dependency of the qualityinformation. E.g. if precision quality for a region of points orconcerning the evaluated orientation of the pole is assessed as beinglow and presented to the user like that, the user may decide to capturemore images related to the respective region and thus providing for moreprecise processing for the region.

Moreover, the quality indication enables a user to capture an adaptedamount of images being necessary for deriving needed data, i.e. the useris informed when enough data is gathered for computing a point cloud ordetermining poses according to measuring requirements to be achieved.Thus, unnecessary gathering of more data than need is suppressed oravoided and the measuring process as a whole is improved with view totime consumption.

A user can continuously check for a level of data-quality to bereachable with already captured images.

Summed up, the fact which is placed at the basis of this qualitydetermination aspect is a possibility to compute an accuracy of the 3Dpoint measured at least on two images obtained using the surveyingsubsystem.

Alternatively to deriving a measurement certainty directly from thecomputation of the position of one or more points (or a point cloud) ofthe surrounding (or additively thereto), a measurement certainty (i.e. agenerally achievable accuracy level) can also be estimated based onintrinsic parameters of the surveying subsystem (like camera resolution,GNSS accuracy) and/or extrinsic parameters (like a distance from the 3Dpoint to the poses of the images, point's observation angle) only(without considering the effective calculation of the point's position).E.g., based on a predefined set of images obtained using the surveyingsubsystem it is possible to compute a field which indicates differentbounds of accuracy for 3D points which are computed using this set ofimages. The field might be coloured with a predefined palette for themore intuitive representation.

On the planning stage (i.e. before capturing the series of images) auser could define an area of interest (e.g. on the available maps orsatellite images). For this area of interest a possible trajectory ofthe surveying subsystem might be automatically computed with a guaranteeof a full coverage of the whole area of interest with target accuracydefined by user. During the survey user could follow this proposedtrajectory to map the whole area with target accuracy without gaps. Thistrajectory might be optimized to minimize its length which reducesmapping time. Any existing objects with known coordinates in the area ofinterest might be taken into account for the trajectory computation.During the survey user can get an estimation of the mapping completenessin real-time. The unmapped parts of the area of interest might be shownto user also in real-time.

On the verification stage it is possible to compute the accuracy fieldfor the actual survey trajectory. Analysis of this field might help auser to check completeness and identify regions where target accuracy isnot reached.

The invention also relates to an automotive surveying system comprisinga position measuring resource, particularly a GNSS-antenna or aretro-reflector, being mounted on a surveying pole and a measuringsubsystem that comprises a camera module and a control and evaluationunit and is adapted to determine positions of the position measuringresource, wherein the surveying pole comprises at least one wheel or achain drive for moving the pole over a ground surface. The camera moduleis designed to be attached to the surveying pole and comprises at leastone camera for capturing images. The control and evaluation unit hasstored a program with program code so as to control and execute alocalizing and mapping functionality in which—when moving along a paththrough a surrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera, the series comprising an amount of images        captured with different poses of the camera, the poses        representing respective positions and orientations of the        camera;    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined,    -   based on the determined poses, a point cloud comprising        3D-positions of points of the surrounding is computed by forward        intersection using the series of images, particularly by using        dense matching algorithm; and/or    -   an orientation of the at least one camera is derived from the        poses for a designated image of the series of images.

In one embodiment of the surveying system, the surveying pole comprisestwo wheels that are arranged on the pole in such a way that a bottom endof the pole is enabled to contact the ground dependent on the pole'sorientation relative to the ground, in particular wherein the bottom endis enabled to contact the ground when the pole is oriented relative tothe ground in a right angle.

In another embodiment of the surveying system, the surveying polecomprises at least one handle, in particular two handles, for allowing auser to push and/or pull the pole along the path.

In a further embodiment, the surveying system comprises a motor fordriving the at least one wheel or the chain drive, in particular

-   -   in response to a pushing or pulling motion by a user and/or    -   for autonomously driving the surveying pole along a predefined        path or driving the surveying pole remotely controlled.

The invention also relates to a two-wheeled, self-balancing motorizedvehicle as part of a surveying system that is adapted to determinepositions of a position measuring resource being mounted on the vehicle,particularly of a GNSS-antenna or of a retro-reflector.

In one embodiment, this vehicle is designed for transporting a user ofthe surveying system, in particular wherein the vehicle is designed insuch a way that the user controls a forward and backward movement of thevehicle by leaning the vehicle relative to a combined centre of gravityof user and vehicle.

In another embodiment, this vehicle comprises measuring point markingmeans for marking an actual measuring point on the ground, in particularwherein the measuring point marking means comprise a laser emittingoptical system for marking the measuring point with a laser spot orpattern.

The invention also relates to a surveying subsystem for such atwo-wheeled, self-balancing motorized vehicle, the surveying subsystemcomprising a camera module and a control and evaluation unit to be usedas part of the surveying system. The camera module is designed to beattached to the vehicle and comprises at least one camera for capturingimages. The control and evaluation unit having stored a program withprogram code so as to control and execute a localizing and mappingfunctionality in which—when moving along a path through a surrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera, the series comprising an amount of images        captured with different poses of the camera, the poses        representing respective positions and orientations of the        camera;    -   a set of image points is identified based on the series of        images, the image points representing reference points of a        reference point field, each image point appearing in at least        two images,    -   the poses for the images are determined based on resection using        the image points; wherein    -   based on the determined poses, a point cloud comprising        3D-positions of points of the surrounding is computed by forward        intersection using the series of images, particularly by using        dense matching algorithm; and/or    -   an orientation of the at least one camera is derived from the        poses for a designated image of the series of images.

The invention also relates to a surveying subsystem comprising a camera,a profiler and a control and evaluation unit to be used as parts of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a hand-carried surveying pole,particularly wherein the position measuring resource comprises aGNSS-antenna or a retro-reflector. The camera module is designed to beattached to the surveying pole and comprises at least one camera forcapturing images. The profiler is designed to be attached to thesurveying pole and adapted for emission of a rotating laser beam as wellas for reception and detection of a returning part of the emitted laserbeam being scattered back from points of a surrounding. The profiler isfurther provided with an electronic distance measuring functionality aswell as an angle measuring functionality for the rotating laser beam sothat profiler measurement data comprising distance and angle informationis gatherable. The control and evaluation unit having stored a programwith program code so as to control and execute a spatial representationgeneration functionality in which—when moving along a path through thesurrounding—

-   -   a series of images of the surrounding is captured with the        camera, the series comprising a plurality of images captured        with different poses of the camera, i.e. from different points        on the path and with different orientations of the camera,    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined,    -   determined positions of the position measuring resource for        points that have been adopted on the path are received by the        control and evaluation unit from the surveying system,    -   a 6-dof-travelling-history including translational and        rotational information in six degrees of freedom is derived for        the moved path based at least on the poses determined within the        SLAM-evaluation and the received determined positions,        particularly by use of a Kalman filter, and    -   coordinates for points of the surrounding are determined as the        spatial representation, based on the profiler measurement data        in combination with the derived 6-dof-travelling-history.

In one embodiment, the surveying subsystem further comprises an inertialmeasurement unit to be attached to the surveying pole and designed forgathering IMU measurement data and the control and evaluation unit isconfigured so that the spatial representation generation functionalityis controlled and executed in such a way, that the6-dof-travelling-history is derived further based on the IMU measurementdata, particularly wherein the IMU measurement data and theSLAM-evaluation are taken into account for deriving the rotationalinformation of the 6-dof-travelling-history, especially wherein the IMUmeasurement data is used to increase resolution of rotational, andparticularly also positional, information derivable from the determinedposes by the SLAM-evaluation, particularly wherein—in case a Kalmanfilter is used for deriving the 6-dof-travelling-history—the Kalmanfilter is fed with the poses determined within the SLAM-evaluation, thereceived determined positions and the IMU measurement data.

In another embodiment of the subsystem,

-   -   the camera module and the profiler, and particularly at least        parts of the control and evaluation unit, are integrated in a        common single housing being attachable to the surveying pole, or    -   each of the camera module and the profiler is integrated in an        own housing, each of the housings being attachable to the        surveying pole, particularly wherein each housing is provided        with mutual connection elements for data transmission, and        particularly also transmission of trigger-information, in such a        way that—in condition of each housing being attached to the        surveying pole—the connection elements are interlocked and a        connection for data transmission, and particularly also        transmission of the trigger-information, between the camera        module and the profiler is provided.

In yet another embodiment of the subsystem, the camera module and theprofiler each comprise a clock and a circuit for assigning a time stampto gathered measurement data like image data, IMU measurement data andprofiler measurement data, particularly wherein for the camera moduleand the profiler a GNSS-module is provided for gathering absolute timeinformation from the received GNSS-signal, so that the GNSS-module isused as the clock.

In a further embodiment of the subsystem, the profiler is designed insuch a way that—in a condition attached to the surveying pole—therotating laser beam defines a laser plane being

-   -   substantially perpendicular to the surveying pole, or    -   inclined with respect to the pole so that they confine an angle        of between about 5 and about 35 degree, particularly of between        about 10 and about 30 degree.

In a further embodiment of the subsystem, the profiler comprises arotatable mirror or a rotatable laser emitting and receiving unit.

In a further embodiment of the subsystem, the profiler is designed in amulti-beam setup in which the profiler is adapted for emission of atleast two rotating laser beams as well as for reception and detection ofa returning part of each of the emitted beams being scattered back frompoints of a surrounding.

In one embodiment of this subsystem, the multi-beam setup is embodiedwith a common rotation platform for the at least two laser beams,particularly wherein the profiler is designed in a three-beam setup. Theprofiler is designed in such a way that the at least two laser beamsdefine laser planes are parallel and slightly shifted with respect toeach other, slightly inclined with respect to each other, particularlyso that slightly different inclination angles with respect to the poleare formed and/or so that slightly different azimuth angles with respectto the pole are formed, or coincident with respect to each other, and/orthe profiler is designed in such a way that the at least two rotatinglaser beams form an angular offset with respect to each other inrotation direction, particularly wherein—in case of a three-beamsetup—the angular offset is 120°.

In one embodiment of this subsystem, the common rotation platformcomprises a common rotatable mirror for rotatably deflecting the atleast two laser beams or the common rotation platform comprises at leasttwo laser emitting and receiving units.

In another embodiment of this subsystem, the multi-beam setup isembodied with at least two rotation platforms for the at least two laserbeams. The profiler with the at least two rotation platforms is designedin such a way that the at least two laser beams define laser planesbeing inclined with respect to each other, particularly so that

-   -   different inclination angles with respect to the pole are        formed, especially wherein one of the laser planes forming an        inclination angle of between about 10 and about 30 degrees and        the other one of the laser planes forming an inclination angles        of between about 60 and about 80 degree and/or    -   different azimuth angles with respect to the pole are formed,        especially wherein an angle offset in azimuth direction of        between about 1 and about 40 degrees is formed between the laser        planes.

In a particular embodiment of the subsystem, the profiler is designed insuch a way that an inclination angle of a laser plane defined by therotating laser beam is adjustable, particularly manually ormotor-driven, so that—in a condition where the profiler is attached tothe surveying pole—the inclination angle is variable with respect to thepole.

In another embodiment of this subsystem, the profiler is designed insuch a way that an azimuth angle of a laser plane defined by therotating laser beam is adjustable, particularly manually ormotor-driven, so that—in a condition where the profiler is attached tothe surveying pole—the azimuth angle is variable with respect to thepole.

In one embodiment of this subsystem, the azimuth angle is adjustable byan actuator. The control and evaluation unit is configured so that thespatial representation generation functionality is controlled andexecuted in such a way, that the azimuth angle is automaticallycontinuously adjusted by the actuator in a manner that alterations ofthe azimuthal orientation of the pole—when moving along the path—arecompensated for, in particular wherein the automatic continuousadjustment of the azimuth angle is based on the derived6-dof-travelling-history or a separate evaluation using

-   -   the poses determined within the SLAM-evaluation, and/or    -   the received determined positions, and/or,    -   in case an inertial measurement unit is provided for, IMU        measurement data.

In particular, higher frequency alterations having low amplitude arecompensated for, especially wherein a relative azimuth angle beingformed between the laser plane and a moving direction on the path iskept substantially constant, particularly perpendicular, in each pointon the path.

In one embodiment of this subsystem, the actual moving direction or asmoothed moving direction is used as the moving direction, the smoothedmoving direction particularly being derived for each point on the pathby smoothing a most recently travelled segment of the path.

In another embodiment of this subsystem, the azimuth angle is adjustedand stabilized with respect to the surrounding by stabilisation meansworking according to the principle of making use of conversation ofangular momentum, particularly wherein the stabilisation means comprisea flywheel, especially wherein the flywheel is rotatable together withthe rotation platform which causes the laser beam to rotate.

The invention also relates to a surveying subsystem comprising a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a surveying pole, particularly of aGNSS-antenna or of a retro-reflector. The camera module is designed tobe attached to the surveying pole and comprises at least one camera forcapturing images.

The control and evaluation unit has stored a program with program codeso as to control and execute a functionality in which—when moving alonga path through a surrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera (31), the series comprising an amount of images        captured with different poses of the camera (31), the poses        representing respective positions and orientations of the camera        (31);    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images with use of a        feature detection process for detecting corresponding features,        particularly by means of feature tracking, especially a        KLT-algorithm (Kanade-Lucas-Tomasi), and/or by means of feature        matching, especially a SIFT-, SURF-, BRISK- or BRIEF-algorithm,        and, based on resection and forward intersection using the        plurality of respectively corresponding image points, a        reference point field is built up comprising a plurality of        reference points of the surrounding, wherein coordinates of the        reference points are derived, and the poses for the images are        determined.

According to this aspect of the invention, the control and evaluationunit comprises several diversified units, wherein at least a first unit,particularly an FPGA or a GPU (graphical processing unit), carries outat least part of the feature detection process for identifying theplurality of respectively corresponding image points, and at leastanother, second unit, particularly a CPU, carries out determining of theposes.

In one embodiment of the subsystem, the first and second units are builttogether as one SoC (system on a chip).

In another embodiment of the subsystem, the first and the second unitare disposed within the camera module.

In a further embodiment of the subsystem, the first unit is disposedwithin the camera module and the second unit is a laptop, a tablet-PC, asmartphone, a processing unit of a handheld controller/datalogger of thesurveying system, a processing unit of a GNSS-module of the surveyingsystem, a processing unit of a surveying station, particularly totalstation, of the surveying system or a surveying site server,particularly being installed in a car.

The invention also relates to a surveying subsystem comprising a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a surveying pole, particularly of aGNSS-antenna or of a retro-reflector. The camera module comprises atleast one camera for capturing images, wherein the camera module isbuilt as one single integrated physical unit being designed to beattached to the surveying pole. The control and evaluation unit hasstored a program with program code so as to control and execute afunctionality in which—when moving along a path through a surrounding—

-   -   a series of images of the surrounding is captured with the at        least one camera, the series comprising an amount of images        captured with different poses of the camera, the poses        representing respective positions and orientations of the        camera,    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined, and    -   a spatial representation, particularly a point cloud, comprising        3d-information about the surrounding is computed by forward        intersection using the series of images and the determined        poses, particularly by using a dense matching algorithm.

According to this aspect of the invention, the control and evaluationunit comprises several diversified units, wherein at least one unit,being integrated in the camera module, carries out at least part of theSLAM-evaluation and at least one other unit, being disposed externallyto the camera module, carries out at least parts of computing of thespatial representation.

In one embodiment of the subsystem, the at least one other unit is acloud server.

In another embodiment of the subsystem, the at least one other unit is alaptop, a tablet-PC, a smartphone, a processing unit of a handheldcontroller/datalogger of the surveying system, a processing unit of aGNSS-module of the surveying system or a surveying site server,particularly being installed in a car.

In yet another embodiment of the subsystem, several other units, beingdisposed externally to the camera module, carry out at least parts ofcomputing of the spatial representation in a decentralised manner, theseveral other units particularly being several smartphones.

In a further embodiment of the subsystem, at least one unit carrying outa part of identifying the plurality of respectively corresponding imagepoints and/or determining the poses is designed so as to be carriable ina backpack of an operator.

In another embodiment of the subsystem, at least one unit carrying out apart of identifying the plurality of respectively corresponding imagepoints and/or determining the poses is a laptop, a tablet-PC, asmartphone, a processing unit of a handheld controller/datalogger of thesurveying system, a processing unit of a GNSS-module of the surveyingsystem or a surveying site server, particularly being installed in acar.

The invention also relates to a surveying subsystem comprising a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a hand-carried surveying pole,particularly wherein the position measuring resource comprises aGNSS-antenna or a retro-reflector. The camera module is designed to beattached to the surveying pole and comprises at least one camera forcapturing images. The control and evaluation unit having stored aprogram with program code so as to control and execute a remote pointmeasurement functionality in which—when moving along a path through asurrounding—

-   -   a series of images of the surrounding is captured with the        camera, the series comprising a plurality of images captured        with different poses of the camera, i.e. from different points        on the path and with different orientations of the camera,    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined, and,    -   upon selection of an image point in an image of the surrounding,        said image basing on at least one of the images of the series of        images, a 3d-position of the remote point being represented by        said selected image point is derived, wherefore a subset of        images related to the remote point is automatically identified        from the series of images and the 3d-position is determined        based on forward intersection using said subset of images and        the determined poses for the images of said subset.

In one embodiment of the subsystem, the control and evaluation unit isconfigured so that the remote point measurement functionality iscontrolled and executed in such a way, that the image point is manuallyselectable by a user, and/or the subset comprises all images in whichthe remote point appears.

The invention also relates to a surveying subsystem comprising a cameramodule and a control and evaluation unit to be used as part of asurveying system that is adapted to determine positions of a positionmeasuring resource being mounted on a hand-carried surveying pole,particularly wherein the position measuring resource comprises aGNSS-antenna or a retro-reflector. The camera module is designed to beattached to the surveying pole and comprises at least one camera forcapturing images. The control and evaluation unit has stored a programwith program code so as to control and execute a functionality inwhich—when moving along a path through a surrounding—

-   -   a series of images of the surrounding is captured with the        camera, the series comprising a plurality of images captured        with different poses of the camera, i.e. from different points        on the path and with different orientations of the camera,    -   a SLAM-evaluation with a defined algorithm using the series of        images is performed, wherein a plurality of respectively        corresponding image points are identified in each of several        sub-groups of images of the series of images and, based on        resection and forward intersection using the plurality of        respectively corresponding image points, a reference point field        is built up comprising a plurality of reference points of the        surrounding, wherein coordinates of the reference points are        derived, and the poses for the images are determined, and    -   de-blurring at least some of the images of the series of images        based on changes in orientation being derived at least dependent        on the determined poses, particularly based on a        camera-trajectory being derived at least dependent on the        determined poses and depth information for the surrounding being        derived at least dependent on the derived coordinates of the        reference points.

In one embodiment of the subsystem, the control and evaluation unit isconfigured so that the functionality is controlled and executed in sucha way, that

-   -   the de-blurring is performed in such a way that motion-blur        appearing in the images due to relative motion of the camera        with respect to the surrounding during exposure-time is        compensated for, wherein a comparatively higher compensation        stage is applied for closer objects of the surrounding being        imaged and a comparatively lower compensation stage is applied        for farther objects of the surrounding being imaged, and/or    -   the camera-trajectory is derived further dependent on determined        positions of the position measuring resource for points that        have been adopted on the path, the positions being receivable by        the control and evaluation unit from the surveying system,        and/or IMU measurement data being gathered by an inertial        measurement unit, which is also provided as part of the        surveying subsystem.

In another embodiment of the subsystem, the control and evaluation unitis configured so that the functionality is controlled and executed insuch a way, that

-   -   the SLAM-evaluation is performed again using the de-blurred        images of the series of images, wherein the poses for the        de-blurred images are re-calculated, and/or    -   a 3d-position of at least one point of the surrounding is        determined appearing in at least some of the de-blurred images,        wherein a subset of images related to the at least one point is        automatically identified from the de-blurred images and        particularly further images of the series of images, and wherein        the 3d-position is determined based on forward intersection        using said subset of images and the determined poses,        therein—where applicable—the re-calculated poses, for the images        of said subset, and/or    -   a point cloud comprising 3D-positions of a plurality of points        of the surrounding is computed by forward intersection using the        de-blurring images and particularly further images of the series        of images as well as the determined poses, therein—where        applicable—the re-calculated poses as the determined poses.

The invention in the following will be described in detail by referringto exemplary embodiments that are accompanied by figures, in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an exemplary embodiment of a surveying system according tothe invention;

FIGS. 2a-c show three exemplary combinations of a camera module and aposition measuring resource as part of a surveying system according tothe invention;

FIGS. 3a-b show a first embodiment of a camera module according to theinvention;

FIGS. 4a-b show a second embodiment of a camera module according to theinvention;

FIGS. 5a-b show a third embodiment of a camera module according to theinvention;

FIG. 6a shows a fourth embodiment of a camera module according to theinvention;

FIG. 6b shows a fifth embodiment of a camera module according to theinvention

FIG. 7a shows a sixth embodiment of a camera module according to theinvention;

FIGS. 7b-c show a seventh embodiment of a camera module according to theinvention;

FIG. 8a shows an eighth embodiment of a camera module according to theinvention having a scanning unit;

FIG. 8b shows a ninth embodiment of a camera module according to theinvention having a scanning unit;

FIG. 8c shows a tenth embodiment of a camera module according to theinvention having a scanning unit;

FIG. 8d shows a eleventh embodiment of a camera module according to theinvention having a scanning unit;

FIG. 8e shows a twelfth embodiment of a camera module according to theinvention having a scanning unit;

FIG. 8f shows a thirteenth embodiment of a camera module according tothe invention having a scanning unit;

FIGS. 9a-b show a fourteenth embodiment of a camera module according tothe invention having a scanning unit;

FIG. 9c shows two laser planes spanned by the rotating laser beams ofthe scanning unit of the camera module shown in FIGS. 9a -b;

FIG. 9d shows a fifteenth embodiment of a camera module according to theinvention having a scanning unit;

FIG. 9e-f show two exemplary embodiments of a scanning unit;

FIGS. 10a-b show a sixteenth embodiment of a camera module according tothe invention having a scanning unit;

FIG. 10c shows a seventeenth embodiment of a camera module according tothe invention having a scanning unit;

FIG. 10d shows the scanning unit of the camera module of FIG. 10 c;

FIG. 10e shows an eighteenth embodiment of a camera module according tothe invention having a scanning unit;

FIG. 11 illustrates a rotation of the camera to the left with a movingobject in the image;

FIGS. 12a-c show varying exposure times for capturing images;

FIGS. 13, 14, 15 a-b show resection based on reference points in asurrounding while moving with the camera module (pole);

FIGS. 16a-b illustrate an exemplary embodiment of a matching of anorthophoto generated by the surveying system and a reference orthophoto;

FIG. 17 illustrate an exemplary embodiment of an orientation determiningfunctionality;

FIG. 18 shows a first exemplary embodiment of a wheeled surveyingsystem;

FIGS. 19a-d show a second exemplary embodiment of a wheeled surveyingsystem; and

FIG. 20 shows a surveying system having a mounted umbrella.

DETAILED DESCRIPTION

FIG. 1 shows an exemplary embodiment of a surveying system 1 accordingto the invention. The depicted surveying system 1 comprises a surveyingpole 10 which is operated by a user 2. The pole 10 comprises a bottomend 11 which is positionable on a measuring point 5 on the ground. AGNSS antenna 15 is placed on the top end of the pole 10 as a positionmeasuring resource of the surveying system 1. Furthermore, the surveyingsystem 1 comprises a camera module 30 and a control and evaluation unit12.

Camera Module

FIGS. 2a-c show camera modules 30,30′ being mounted on a pole 10together with a position measuring resource of the respective surveyingsystem.

Each camera module 30,30′ comprises an optical recording device 31 thatis sensitive to light coming from all or many spatial directions. Itcould be based on an imaging sensor and a fish-eye lens, or acombination of a camera and a parabolic mirror, or a minimum of twosingle cameras arranged on a horizontal ring, or any other optical setupfunctioning as a wide-angle or panorama camera.

The camera module can be a separate module 30 which is mounted on a pole10 together with a GNSS antenna 15 (FIG. 2a ) or a reflector 16 (FIG. 2b). Moreover, the module 30′ can be integrated into the housing of a GNSSantenna (FIG. 2c ) or reflector.

According to FIG. 2c the camera module 30′ is represented by theposition measuring resource which additionally comprises a GNSS antennaand/or a reflector.

The FIGS. 3a and 3b show a first embodiment of a camera module 30according to the invention. The camera module 30 has a housing 40 andmounts for the pole 38 and for the position measuring resource 39 (GNSSantenna or reflector). It may comprise a set of cameras 31, e. g. foursingle cameras 31 aligned in angles of 90° to each other with ahorizontal field-of-view >90°. In such an arrangement a horizontalfield-of-view of 360° is covered. The vertical field-of-view (FOV) ofthe camera assembly can be about 60°.

The cameras can be aligned horizontally or downward oriented, e. g. by20°, as shown in FIG. 3b . This is advantageous for applications whereclose objects are of particular interest.

Moreover, a processing unit 33 can be part of the camera module 30. Theprocessing unit 33 can be a CPU, e. g. an ARM processor or a combinationof a CPU with an FPGA, e. g. Zync SoC, or a combination of a CPU with agraphical-processing-unit (GPU). In case of a combined processing unit,e. g. feature tracking, etc. is carried out on the FPGA or the GPU.These are primarily image processing algorithms where a high degree ofparallel processing can be achieved on units of that kind.

Also an inertial-measurement-unit (IMU) 34 can be part of the cameramodule. The IMU 34 may consist of a 3-axis accelerometer and,particularly, of a 3-axis gyroscope.

Additionally, a magnetometer may be included in the IMU.

The FIGS. 4a and 4b show a second embodiment of a camera module 30according to the invention: In order to increase the vertical FOV of thecamera module 30, four downward-oriented cameras 31′ can be combinedwith four upward oriented cameras 31″.

Alternative embodiments are shown in FIGS. 5a and 5b : FIG. 5a shows anarrangement with two fish-eye cameras 36 with a horizontal field-of-viewof >180°, and FIG. 5b shows an arrangement of a single camera 31 with amirror 32, particularly a parabolic mirror, and a glass window 35.

Optionally, the cameras 31,31′,31″,36 of the camera modules 30 describedabove can have different resolutions. For instance, in case the cameramodule 30 comprises eight cameras, four cameras can have low resolutionand are read-out in high frame rate (advantageous for feature tracking)and four cameras have high resolution (advantageous for dense matching)and are read-out with a lower frame rate. Alternatively, the highresolution cameras are not running with a specific frame rate but aretriggered by the algorithm when a keyframe should be captured, e. g. ina distance interval of two meters.

Alternatively or additionally, according to one aspect of the invention,one single camera of the camera module is built so that image capturingwith at least two different resolutions and/or different frame rates isprovided by the camera. Thus, high-resolution images as well aslow-resolution images can be provided commonly with view to a measuringprocess.

Moreover, for reducing reflections from windows, a polarization filtercan be mounted in front of the camera lenses (not shown here).

The FIGS. 6a and 6b show a two further embodiments of a camera module 30having absorber means 37,37′. The relative position and orientation ofthe cameras 31 can be determined in a calibration procedure. In order tokeep the alignment of the cameras 31 stable over time, e. g. having itresistant against shocks or drops, an absorber 37 can be integrated, e.g. between the housing 40 and the mount for the pole 38, as shown inFIG. 6a . Alternatively, as shown in FIG. 6b , the cameras 31 and theIMU 34 can be mounted on a frame 41 which is mechnically decoupled fromthe housing 40 by absorbers 37,37′. The absorbers 37 also reduce themaximum accelerations and avoid the accellerometers to saturate.

Such damping element (absorber) could be in form of a gimbal mounting orthe like.

Trigger

During the measurement, image data is recorded with the camera module30. In case that there is more than one camera 31 in the camera module30, the image data is recorded in parallel, i.e. synchronously. Thetrigger signal for triggering all the single cameras can be produced byanother sensor, e. g. the GNSS antenna. Alternatively, one camera cantrigger all the others.

Image data can be recorded as a video with about 15 to 25 frames persecond (FPS) or as a set of event-based triggered still images, e. g. animage is recorded when the operator moved two meters since the lastimage was taken. Alternatively, a new image can be captured, when theimage content shows a significant difference to the previous image.

Moreover, the recording of an image can be triggered by the operator,when he places the pole on a point and triggers a measurement.

FIGS. 7a-c show two further embodiments of the camera module 30. Eachcamera module 30 comprises at least two cameras 31 a-d and opticaltriggering means 70,71 for triggering a highly synchronous capturing ofimages by the cameras 31 a-d. This is useful for instance to allow animage stitching of the images captured by the single cameras even if thecamera module is in motion.

In FIG. 7a , a ring of flash lights 70 is arranged around the cameramodule 30, here around the mount for the position measuring resource 39.The cameras 31 a,31 b are adapted for perceiving a light flash from theflash lights 70 in the surrounding. The cameras 31 a,31 b then cansynchronously capture an image. Furthermore, synchronously capturedimages of the cameras 31 a,31 b can be identified by the means of thelight flash that appears in all images that have been captured duringthe flash.

FIGS. 7b and 7c show a camera module 30 with four cameras 31 a-d. Thecamera module 30 comprises optical triggering means designed as groupsof light emitting diodes (LED) 71. These are arranged in such a way thateach group 71 lies in the field-of-view (represented by dashed lines) oftwo of the cameras 31 a-d and is perceivable by these two cameras. Thegroups of LED 71 can be used to trigger a highly synchronous capturingof images by the cameras 31 a-d. The groups of LED 71 can also be usedto add a code to the images, e. g. in order to allow identification ofsynchronously captured images.

Scanning Unit (→Profiler) [Section Still Being to be Complemented by theWording of the Corresponding Claims]

In the FIGS. 8a-f, 9a-f and 10a-e various exemplary embodiments ofcamera modules 30 are depicted that comprise a scanning unit 50.

The scanning units 50 integrated in the camera module 30 areadvantageous for the generation of point clouds in real-time, i.e. noexpensive dense matching step has to be performed as in the case ofpoint cloud generation with images. Moreover, in contrast to camerabased approaches the scanning unit 50 does not rely on good texturedsurfaces for the derivation of point clouds.

The scanning unit 50 can consist of a laser emitting and receiving unit51 and a rotation mirror 52. In the arrangement shown in FIG. 8a therotating laser beam 55 is spanning a more or less horizontal plane, ifthe pole is aligned more or less vertically.

In FIG. 8b another setup of the scanning unit 50 is shown. Here, therotation axis of the mirror is tilted by about 10° to 30° from thehorizontal plane.

FIG. 8c shows the camera module 30 of FIG. 8b mounted on a surveyingpole 10.

The rotating laser 55 spans a laser plane which is tilted in such a waythat it passes the GNSS antenna 15, and, consequently, the occlusionsare small. For such a setup where the camera module 30 is mounted closeto the GNSS antenna 15 on the top of the pole 10, the scanning modulehas to be somehow exposed. To avoid damages of the scanning unit 50 whenthe pole 10 is dropped, preferably a drop protection 18 is mounted belowthe camera module 30 on the pole 10.

FIG. 8d shows a combination of a camera module 30 and a separatescanning module 58. Both modules 30,58 can be plugged together. Forenergy supply and data transfer a connector 59 can be integrated intothe housing of each module 30,58. The scanning module 58 can be equippedwith a separate processing unit 53. Alternatively, the computations ofthe scanning module 58 can be carried out on the processing unit 33 ofthe camera module 30.

FIG. 8e shows an integrated camera module 30 with a scanning unit 50′having a rotating laser emitting and receiving unit 51′ instead of arotating mirror.

FIG. 8f shows a camera module 30 with a scanning unit 50″ having amulti-beam setup with three laser emitting and receiving units 51′rotating together around one axis.

Alternatively, instead of emitting three parallel laser beams55,55′,55″, the laser emitting and receiving unit can be mounted with anangular offset of 120°, e. g. like the blades of a wind turbine (notshown here).

FIGS. 9a and 9b show an arrangement with two scanning units 50 a,50 b.The rotating beams 55 a,55 b of both of them span two tilted laserplanes 56,56 a (FIG. 9c ).

The tilt angle influences the scanning resolution on the object, i.e.the density of the point cloud. On the one hand, an almost verticalplane scans the nearby ground in a (probably too) high resolution sincethe lever arm with about 2 m is quite short. Moreover, quite many raysare “wasted” since they are aimed to the sky. On the other hand, analmost horizontal plane does not cover the sky and the nearby ground atall. A combination of two scanning units where the rotation axis of onemirror is tilted by about 10° to 30° from the horizontal plane and theaxis of the second mirror is tilted by about 10° to 30° from thevertical plane could lead to a improved distribution of the scannedpoints.

Alternatively, the vertical angle of the laser plane can be madeadjustable (this is shown in FIGS. 10a and 10b ). This enables the userto change the tilt angle according to his specific needs. The rotationcan be continuous or there can be some predefined angles for high,medium and low. In the latter the scanning module clicks into thepredefined positions.

Another problem that might appear in practice is the rotation of an(almost) vertical plane. In case the user walks with a constant velocityof 1 m/s in the direction of the rotation axis of the mirror andassuming a rotation rate of the mirror of 50 Hz, i.e. 50 revolutions persecond the offset of neighbouring tracks on an object in a distance of15 m are 2 cm. However, if the pole is rotated about the vertical axiswith a rotation rate of 30°/s the offset between neighbouring tracks isabout 16 cm. Consequently, such rotations whether intented or not maylead to an inhomogeneous point distribution.

In order to overcome this problem, as shown in FIGS. 9a and 9b , twoscanning units 50 a,50 b which span almost vertical planes can beapplied in such a way that there is a small horizontal angular offset ofabout 5 degrees.

An alternative embodiment is depicted in FIGS. 9d-f : The scanning unit50′″ comprises a plurality of laser emitting and receiving units 51.Thus, instead of one laser plane a fan of either diverging (FIG. 9e ) orparallel (FIG. 9f ) laser planes is generated.

Alternatively, as shown in FIGS. 10a and 10b , the scanning unit 50′″″can be integrated into the camera module 30 in such a way, that anunintended rotation of the pole can be compensated by a rotation of thescanning unit 50′″″ into the opposite direction, in particular actuatedby a drive 57. The rotation angle can be derived from measurements withthe camera module 30, e. g. the angular rates from the IMU 34. The aimof the compensation can be that the azimuth of the laser plane isconstant independently from the movements of the user.

Alternatively, with such a mechanism the scanning plane canautomatically be aligned orthogonally to the walking direction of theuser. The direction can for instance be derived from the state vector ofthe Kalman filter. There is the advantage that in case the user walksaround a curve the plane follows this movement automatically.

Alternatively, instead of rotating the whole scanning module, only theray can be redirected by an additional oscillating mirror 52′. This isshown in FIGS. 10c and 10 d.

Alternatively, as shown in FIG. 10e , the stabilization of the laserplane, i.e. to keep the azimuth angle almost constant (smooth quickmovements), can be also achieved by implementing the gyro principle.Here, a flying wheel 52″ rotates together with the mirror around theaxis to be stabilized.

Moving Objects

In FIG. 11, a rotation of the camera to the left is illustrated with amoving object in the image. During data acquisition also masking ofdifferent areas can be applied in order to remove undesired (here:moving) objects. This for instance include static objects relative tothe pole like the surveyor itself as well as moving objects likepedestrians, cars and other non-static/relative static objects.

The masking can be done semi-automatically with user interaction orfully automatically.

A possible scenario for user interaction guided processing is theexclusion of the surveyor in tracking feature points on the images,where the user, for instance, roughly masks the outline of itself on theimage and its silhouette is recognized and subsequently tracked by anappropriate algorithm (e. g. standard segmentation algorithms, activecontours or template matching).

A fully automated algorithm might detect interfering objects based onsome motion estimation algorithms, e. g. optical flow, and reject thecorresponding tracked features.

For instance, as is shown in FIG. 11, a rotation of the camera to theleft would result in an optical flow indicated by the displayed arrows111. Although becoming more complex by more advanced movements of thecamera, this vector field (optical flow) can be used to determine movingobjects—indicated by the displayed arrows 112—within the image (e. g. byanalyzing discontinuities or anomalies like inconsistent local changes)or even support segmentation or other image processing relatedalgorithms.

Possible variants of different procedures might include interferingobject detection independently on each image, initial detection andtracking, as well as global motion elimination. Additionally, in orderto become more robust, the final algorithm for removing moving objects(or corresponding inconsistent feature points) might consist of amulti-step procedure, e. g. local and global motion estimation. Sinceglobal motion in this context refers to camera movement, there mightalso be some additional sensor information (e. g. GNSS or IMU data)used, to predict global movement and, subsequently, to stabilize thedetection of moving objects.

The detection of corresponding (interfering) features can be carried outon the processing unit in the camera module. Particularly, if an FPGA isincluded in the processing unit parts of the feature detection can becomputed very efficiently on the FPGA.

After the identification of corresponding features the poses of theimages, i.e. position and orientation, are computed. This can be donefor every image with a sufficiently large number of detected pointfeatures. However, particularly if the image data is recorded with highframe rate processing power can be saved by selecting a subset of imagesand determine the pose only for those selected images.

A criterion for the image selection can be the baseline, i.e. thedistance between the current image and the previous one, e. g. adistance of one meter. Small baselines result in a bad accuracy of 3Dpoints determined by forward intersection. Another criterion can be theimage quality, i.e. only image with good quality (high sharpness, nounder- or over-exposure, etc.) are selected.

Alternatively, the image selection can be based on a combination ofimage data, IMU data and GNSS data or any subset of these data sources.For example, the IMU could be used to select images that don't sufferfrom motion blur by considering the movement and especially the rotationduring exposure. This has the advantage that blur-free images areselected even when the image content changes significantly. Differentfilters (e. g. baseline and motion blur) can be combined to achieve anoptimal image selection leading to an optimal reconstruction of thescene using limited computational resources.

In case there is a combination of low resolution and high resolutioncameras the low resolution cameras are mainly used for feature trackingin a high frame rate. The high resolution cameras need not to captureimages in a defined frame rate, but can be triggered at specific times,e. g. low movement, e. g. sensed with IMU, or distance between currentand last image (=baseline) of for instance two meters.

Moreover, images can be taken with varying exposure times, e. g. theimages are taken with exposure times of 1 ms, 2 ms, 4 ms, 8 ms, 10 ms,20 ms and so on. This is illustrated in FIGS. 12a-c . In a definedwindow around the optimal time for capturing a key frame the frame withgood exposure is selected. In this case, as shown in FIG. 12a , theimage acquisition interval can be regular, i.e. an image in an intervalof 8 ms.

Alternatively, as shown in FIGS. 12b and 12c , the image captureinterval can be irregular.

Moreover, a HDR (high dynamic range) image can be generated based on aset of images captured with different exposure times. This can be donewhen the camera is held sufficiently still, which can be sensed with theIMU or GNSS or camera data or a combination of all. Alternatively, thecamera can be moved and the offset determined with thestructure-from-motion algorithm is considered in the generation of theHDR image.

In order to ensure data sets that can be processed, there may be somedirect user feedback during data acquisition. This feedback may includelight indication (e. g. status LEDs), audio signals or force feedback(vibrating pole, smartwatches, vibration wristlet, smartphone, tablet,etc.). There might also be the option to directly visualize the currentstate of data recording, e.g. on a mobile phone, tablet or specialglasses (interactive glasses).

Algorithm for Determination of Poses and Simultaneous Generation ofSparse Point Cloud

For the derivation of a pole's six degrees of freedom, i.e. the positionof the module and the orientation angles (roll, pitch, yaw) with respectto an outer/external (e.g. global) coordinate system a SLAM(simultaneous localization and mapping) algorithm can be applied.

The determination of the 6-DoF is based on measurements from the cameraand optionally additionally position measurements from the GNSS system(and/or total station with retro-reflector). Moreover, accelerations andangular rates measured with the IMU can also be included into thedetermination of the poses (i.e. position and orientation with 6 degreeof freedom; i.e. 6-dof).

The image data is analyzed for corresponding features (orcorresponding/homologue image points), i.e. the position of thecorresponding images of one identical object point 61 in several images.This is done using feature detection and matching algorithms such asSIFT, SURF, BRISK, BRIEF, etc. By identifying several correspondingfeatures (or corresponding image points, also called homologue imagepoints), several object points 61 are determined. These object pointsbuild up a reference point field, which can be used for eachadditionally gathered camera image as a reference, so that in each newlyadded image existing points of the reference point filed can be used toreference the image with respect to all previous images.

Hence, in each newly added image, again corresponding image points (inthe added image and already previously existing images) arefound/identified for object points of the already existing referencepoint field. And these found/identified image points in the newly addedimage are used (together with the previously determined coordinates ofthe according object points being represented by the found/identifiedimage points) to determine the pose of this newly added image, by meansof resection.

Furthermore, also in each newly added image again corresponding imagepoints (in this added image and already previously existing images) arefound/identified for new object points of the surrounding. And thesefound/identified corresponding image points in the newly added image andat least one “old” image (i.e. the positions of the corresponding imagepoints in the newly added image and the old image) are used to determinecoordinates of further object points by forward intersection, so thatthe reference point field is expanded therewith by the further objectpoints. Hence, on one hand, for each newly added image the pose can bedetermined (based on resection using existing feature/image pointschemes of existing object points of the already existing referencepoint field) and, on the other hand (logically simultaneously), witheach newly added image the reference point field is also growing (basedon forward intersection using newly identified correspondingfeatures/image points of previously unknown object points of thesurrounding).

Alternatively, in case of a video input, the correspondences (i.e. thecorresponding or homologue image points or features) can be found usinga tracking algorithm on each video frame, e.g. by applying theKanade-Lucas-Tomasi (KLT) feature tracker. In the sense of thisapplication, the term of “capturing a series of images with one or morecameras” includes generating image data by consecutively (e.g. followinga defined timing, like one image per second, or following a definedspacing, like one image every half meter of movement) capturing/takingsingle images or by capturing a video frame.

Hence, in the SLAM-evaluation based on the corresponding features,simultaneously the poses of the images are derived and a point cloud(herein called “reference point field”, which is a point cloud withcomparatively low density (therefore also often called “sparse pointcloud”)) is built up and computed. This reference point field is atfirst instance only computed so as to determine the poses of the images(i.e. so as to determine and reference the positions and orientations ofthe image-pickups relative to each other), i.e. at least initially onlyfor localisation purposes. The reference point field, thus, is aconsecutively growing common reference frame for determining the poses.The reference point field usually does not comprise pre-known points ofthe surrounding.

In a first step of the reference-point-field- and poses-determination, arelative pose algorithm is used to calculate the relative pose betweentwo selected initial frames/images and an initial point cloud (i.e. afirst part of the reference point field).

Therein, IMU and/or GNSS data can be used additionally to determine therelative pose of the selected initial frames (so as to make the step ofdetermining the poses more stable and/or more efficient). On the basisof the resulting point cloud (this first/already existing part of thereference point field) and the corresponding features detected in athird image the pose of the third image can be computed by resection.Then again, forward intersection is applied to refine the 3D coordinatesof the features detected in all three images and to determine new 3Dpoints which are detected in one of the first images and the third one.Consequently, with added image new 3D points might be reconstructed andthe number of points in the point cloud (reference point field) isgrowing (FIG. 13).

Summary for the basic part of the algorithm:

Simultaneously

-   -   poses for the images are determined and    -   a reference point field is built up comprising a plurality of        reference points existing in the surrounding        by applying defined algorithm to the series of images (13),        wherein    -   a plurality of respectively corresponding image points are        identified in each of several image sub-groups, each image        sub-group containing at least two images of the series of        images,    -   the poses are determined by means of resection using determined        coordinates for reference points already added to the reference        points field and image-positions of the corresponding image        points representing said reference points and    -   reference points—each thereof being represented by identified        corresponding image points—are added to the reference point        field and coordinates for the reference points are determined by        means of forward intersection using the image-positions of the        corresponding image points and the determined poses for the        images in which said respective corresponding image points        appear.

Even further summed-up (with slightly other words):

A SLAM-evaluation with a defined algorithm using the series of images isperformed, wherein a plurality of respectively corresponding imagepoints are identified in each of several sub-groups of images of theseries of images and, based on resection and forward intersection usingthe plurality of respectively corresponding image points,

-   -   a reference point field is built up comprising a plurality of        reference points of the surrounding, wherein coordinates of the        reference points are derived (basically from the forward        intersection) and    -   the poses for the images are determined (basically from the        resection).

As a final or intermediate step (i.e. after the basic algorithm has beencarried out or in addition thereto), the overall solution (i.e. thebuilt up reference point field (i.e. the derived coordinates of thereference points) and the determined poses, and optionally also thedetermined image-positions of the corresponding image points in theimages), can be refined using bundle-adjustment. This algorithm is anon-linear least squares minimization of the re-projection error andoptionally also the GNSS measurements. It will optimize the location andorientation of all camera poses and all 3D points of the reference pointfield in one step.

Bundle-adjustment can be triggered by the user when he is using the poleto measure individual points, i.e. in traditional GNSS pole mode.Bundle-adjustment is then used to refine the location and orientation ofthe pole during the measurement to provide the user with an optimalestimate for the position of the tip during the measurement.

Alternatively, image data, IMU data and/or GNSS data can be used todetect if the pole is held steady in one position. Bundle-adjustment isthen triggered automatically. When an optimal—bundle-adjusted—result isavailable this is indicated to the user through e.g. a green light. Theuser can then immediately read out an optimal estimate for the positionof the tip upon triggering a measurement.

If GNSS data is available—which might not always be the case—theposition can be used in the resection of a new camera pose or in thecomputation of the relative pose the initial image pair. For thecomputation of a new image pose the detected feature positions in theimage and the GNSS position are combined in the resection algorithm.

Particularly, the accuracies of the measurements, e.g. 1 px for thefeature measurements and 3 cm for GNSS positions, are considered and themeasurements are weighted accordingly.

The combination with GNSS data leads to the generation of ageo-referenced and scaled point cloud by the SLAM algorithm. Moreover,since GNSS delivers positions with good accuracy on a global scale, thepositions counteract the error accumulation and the resulting driftwhich might appear in SLAM based on image measurements only.

However, if a basic point cloud is already generated, the resection alsoworks if no GNSS signal is available by image data only. This might bethe case when the operator stands very close to a façade and most GNSSsatellites are covered by the building. The 6-DoF are then derived byresection of the camera pose based on the existing point cloud.

Alternatively, other SLAM or Structure from Motion algorithms can beused for the 6-DoF determination of the pole.

Moreover, if no GNSS position is available and the position is derivedfrom image data, particularly in combination with IMU data, the derivedposition can be feed back to the GNSS module for a fast reacquisition ofthe fix for the GNSS position.

Alternatively, when applying the camera module 30 shown in FIG. 2b to areflector 16, instead of GNSS data also positions measured with a totalstation can be used. Here again, if a basic point cloud is alreadygenerated, the pose of the module can be determined also when theline-of-sight between total station and reflector is interrupted.

In the combination of GNSS data with image measurements the offsetbetween the antenna centre and the camera module has to be considered.This offset might be known from the mechanical design or derived bycalibration. When multiple cameras are used, their relative poses haveto be calibrated or derived from the design as well. Calibration can bedone by the manufacturer or by the user in a specially designedprocedure using e.g. a well textured area (reference pattern) beingcapturable be the camera to be calibrated. A defined relative spatialrelationship between the pattern and the camera is pre-known. Thecalibration can also be fine-tuned in the bundle adjustment(self-calibration).

Furthermore, the data from the inertial measurement unit (IMU), i.e.accelerations and angular rates, can be included in the determination ofthe six degrees of freedom. Since the accelerometer senses the gravityvector, the tilt of the camera module, i.e. the roll and pitch angle,can be directly determined from the IMU data. The horizontal orientationof the pole, i.e. the yaw angle, can be derived from a comparison of theposition offset based on the double integration of accelerations withthe position offset derived from the GNSS positions. Alternatively oradditionally, a magnetometer could be used to determine the yaw angle.

The combination of the sensor data can be carried out by a sensor fusionalgorithm, e. g. a Kalman filter, a particle filter etc.

For the sensor fusion it is important that all the measurement datarefer to a common time basis. This can be achieved by a synchronizeddata acquisition, i.e. one sensor triggers all the others, or byassigning a time stamp to each measurement.

Knowing the length of the GNSS pole and its pose with respect to asuperior, either local or global coordinate system, e.g. WGS84, thecoordinates of the tip point can be derived. In practical surveying thishas the advantage that the operator does not have to level the poleanymore after he places the tip point onto the measuring point. Becauseof the reduced time needed for measuring a single point, this has apositive impact on the productivity of the operator performing themeasurement job.

Dense Matching

Dense matching has the goal to determine a dense point cloud, i.e. a3D-coordinate for each pixel or a subset, e.g. on a regular 3×3 grid,i.e. for every 3rd pixel in row and column direction, in the originalimages. The algorithm consists of two major steps.

First, for all overlapping cameras a disparity map is computed. This mapcontains the offset of a pixel in two images, i.e. the shift to beapplied to a pixel in the first image to end up at the position of thecorresponding point in the second image. There are multiple ways tocompute these maps, correlation techniques, e. g. Semi-Global-Matching,etc.

For each pixel in the disparity map a confidence value can be obtainedand used in the further processing, e. g. for adaptive weighting of themeasurements.

Second, using this set of disparity maps 3D points are computed byforward intersection.

Knowing the pose of the image rays 60 from the projection centersthrough the corresponding pixels are set up. The 3D coordinates of theobject point results from the intersection of these rays. In principle aminimum of two rays is needed for the intersection of a 3D point.However, in practice as many rays as available are used in the forwardintersection.

Identifying the image points (reference points) and determining posesfor images based on the identified image points (particularly computinga point cloud based thereon) is a be understood as a form or at leastpart of a dense matching process.

For the forward intersection least squares adjustment can be applied.Here, the 3D coordinates of the points are determined by minimizing thedeviations—actually the squared deviations—between the point and all therays. Based on the geometry and the resulting deviations, i.e. theremaining residuals, the quality of the intersection can be derived bycomputing the variance-covariance matrix and, furthermore, an estimatefor the standard deviation of all three coordinates of the 3D point.

This is shown in FIG. 15a , where a “good” intersection 66 of six raysleads to a small uncertainty and a “bad” intersection 68 of three raysleads to a large uncertainty, particularly for a single point or arespective region of points.

A quality indicative output concerning the quality of at least one ofthe computed points of the point cloud as described above may be basedon such computation.

The final point cloud can be filtered using several criteria on themeasurement quality.

This includes the number of images where the point is observed, i.e. thenumber of rays used for the intersection, the baseline between thecameras which defines the geometry of the intersection, a measure forthe consistency of all the measurements, etc.

Alternatively, a quality indicator can also be computed for an actuallytravelled path and actually acquired series of imaged (and also for aplanned path with a planned capturing of images) solely base onpre-known circumstances like intrinsic factors (camera resolution, etc.)and general extrinsic factors (distance to camera, etc). In this case, acomputation for the position of at least one point (and a considerationof a degree of intersection of the rays within the forward intersection)would not be necessary to for this estimated quality indicator. FIG. 15bshows a visualisation of accuracy bounds (i.e. an accuracy map) based onan estimated reachable quality for the determination of 3d-positions inthese regions. Points lying close to the planned or travelled trajectoryhave a higher estimated accuracy/quality and points of the surroundinglying farther from the planned or travelled trajectory have a lowerestimated accuracy/quality.

Alternatively, other algorithms, e.g. plane sweeping, can be used todetermine the dense point cloud.

For the measurement of a single point the operator places the tip pointof the pole on the measuring point. Contrary to this, the measurement ofa dense point cloud is dynamic, i.e. the operator just walks aroundcollecting data, i.e. image data, position data from GNSS or a totalstation, IMU data, etc.

The user simply walks through the area to be mapped. During the movementthe system records the image data, e.g. 15 frames per second, the GNSSpositions and the IMU data.

The sensors are either synchronized, e.g. triggered by a master sensor,or a time stamp is assigned to each measurement.

De-Blurring

A further inventive aspect is to de-blur images of the acquired seriesof images after computing “SLAM” (or as also called “Structure fromMotion SfM”).

It is known that—in images taken with a camera being in motion—a motionblur may appear in the images. Therein, in general, objects of theimaged scene/surrounding being farther from the camera have a lowerrelative motion with respect to the camera (i.e. the relative motionbeing effective for the image-pick-up) and objects of the imagedscene/surrounding being closer to the camera have a higher relativemotion.

From the SLAM algorithm (particularly additionally with the aid of IMUdata and/or GNSS/TPS position measurement data), a 3d-trajectory for thecamera can be derived for the path travelled during acquisition of theseries of images.

Also, a sparse point cloud for the scene/surrounding (i.e. the positionsof the reference point field for the surrounding extracted and used toderive the poses for the images) is also already determined from theSLAM algorithm.

Based on this sparse point cloud (i.e. more generally spoken: depthinformation or a depth map for the scene/surrounding basing on thedetermined sparse point cloud) and based on the motion of the cameraduring acquisition of each of the images, a motion de-blurring can beperformed in the images by image processing.

Therein, imaged objects having been closer to the camera duringacquisition of the image are applied with a comparatively higherdegree/stage of motion de-blurring and imaged objects having beenfarther from the camera during acquisition of the image are applied witha comparatively lower degree/stage of motion de-blurring.

After de-blurring of images of the acquired series of images, the newlygenerated imaged by de-blurring, which can be referred to as de-blurredimages, can then substitute/replace the corresponding un-de-blurredimages of the series of images.

In case all images of the series of images are de-blurred, thesede-blurred images can then constitute the series of images (i.e. thencomprising only de-blurred images).

In case only a portion of the images of the series of images isde-blurred (substituting—in the series—the corresponding/counterpartun-de-blurred images belonging to the series before de-blurring), theseries of images can then consist of de-blurred images and un-de-blurredimages.

In the following, these newly or refreshed series of images thencontaining at least also some de-blurred images (or only de-blurredimages) can be used for several purposes:

a) re-performing the SLAM-evaluation by using the refreshed series ofimages (comprising de-blurred images), i.e. resulting in

-   -   re-calculated positions for the reference points of the        reference point field (i.e. a re-calculated sparse point cloud)        and    -   re-calculated poses for the images.

Therein, due to the fact the features to be identified in the images,which represent the reference points, can be located in the images withhigher accuracy/higher robustness and less measurement uncertainty (dueto lower degree of blur in the images after de-blurring), both thepositions of the reference points and the poses for the images can bedetermined more accurate and with less measurement uncertainty.

b) calculating a dense point cloud for the surrounding (i.e. performingthe dense reconstruction or meshing) with higher accuracy and lessmeasurement uncertainty.

Therein, either the refreshed/de-blurred series of images (comprisingde-blurred images) can directly be used for the dense reconstruction. Orthe step described under point a) above (i.e. re-performing theSLAM-evaluation with the refreshed series) can be prefixed andthen—basing on the already re-calculated poses by performing step a)above—the dense reconstruction can be performed using the de-blurredseries and the already more accurately determined poses.

This results in a higher accuracy for the determination of the densepoint cloud due to the fact that in de-blurred images the identificationand localisation of corresponding image points in the images (belongingto identical scene points) can be performed with higher robustness andhigher precision. Hence, also the forward intersection using theidentified and localized image points reveals the position of the scenepoint with higher accuracy/less measurement uncertainty.

Also, a higher density for the determination of the dense point cloud isachievable due to the fact that in de-blurred images more correspondingimage points in the images (belonging to identical scene points) areidentifiable.

c) calculating a position of single points of the surrounding(hereinbefore and -after also called “remote point measurement”) withhigher accuracy and less measurement uncertainty.

Analogue to the calculation of a plurality of points (i.e. a dense pointcloud for the surrounding), also positions for single points can bedetermined with higher accuracy and less measurement uncertainty byusing the refreshed/de-blurred series of images (and optionally also thealready re-calculated poses).

Summed up, using de-blurred images, SfM-/SLAM-evaluation on de-blurredimages can provide for better results regarding the determination of theposes and the sparse point cloud, as features will be located withhigher precision (and more features may be located) due to the highercontrast in the images. If the SfM is more precise, then further stepswill provide better quality (dense reconstruction, meshing). But alsowithout a refreshed SfM-/SLAM-evaluation in prefix, the densereconstruction can be performed with at least somewhat higher quality byusing the de-blurred images compared to the scenario of usingun-de-blurred images.

Orthophoto

In the FIGS. 16a and 16b an exemplary embodiment of a matching of twoorthophotos is illustrated, wherein one orthophoto 160 is generated bythe surveying system according to the invention and one is a referenceorthophoto 165, for instance an aerial image. The term orthophoto hereis understood as meaning a “true” orthophoto having an orthographicview, i.e. relief and tilt have been adjusted in order to orthorectifythe image.

Since the cameras may be looking downwards, or have a sufficient fieldof view, the ground plane will be visible. Together with a knowndistance to the ground, camera locations and orientations, the imagescan be projected to the ground plane. By using many images fromdifferent directions, one large composite image of the ground plane canbe computed. Due to the texture foreshortening (perspective effect, whencameras are looking at a certain angle to the ground), the methoddescribed in the paper “Analysis-by-Synthesis Texture Reconstruction”(Liefers, Parys, Schilling, 2012) can be used to obtain a high qualitytexture for the ground plane. The generated orthophoto can be registeredto a georeferenced aerial image in order to localize the measurements inthe geographic coordinate system. In the following paragraphs, theworkflow is described in detail.

The processing of the camera poses and the dense matching, i.e. thegeneration of the dense point cloud can be carried out on the processingunit as part of the camera module.

Alternatively, the processing can be carried out on a controller (datalogger) or the processing unit of the GNSS antenna which are connectedto the camera module, either by a cable or via radio, Bluetooth, WiFi,etc.

Moreover, the data can be transmitted to a dedicated cloud server whichis connected to the internet. The data can be transmitted directly fromthe camera module or via the controller or via the GNSS antenna.

Alternatively, the server can be installed in a vehicle, e. g. a carwhich is located close to the surveying area and communicate with thepole through a local telecommunications protocol such as Bluetooth orWiFi.

Alternatively, the server can be temporarily or permanently installed onsite e. g. in a construction shed and communicate with the pole througha local telecommunications protocol.

Preferably, the transmission of the data and the processing on the cloudserver starts immediately after the recording is started. In this casethe processing is carried out in parallel to the data acquisition in thebackground which helps to keep the latency of the result short.

Alternatively, the system can include a second processing unit which istogether with a power supply, e. g. batteries, carried by the operatorin a backpack. The second processing unit may be equipped with agraphical processing unit (GPU) which enables massive parallelprocessing of the image data. Particularly, the second processing unitcan be a standard portable device such as a powerful laptop, tabletcomputer or smartphone. The second processing unit may communicate withthe pole through cable or a local telecommunications protocol.

Moreover, a combination of processing on the processing unit of thecamera module, the processing unit of the controller or the GNSS antennaand external processing units such as a cloud server. For instance, thesynchronized recording of the data can be carried out on the processingunit included in the camera module. Also, a pre-processing of the imagedata, e. g. the feature extraction or some basic image processingalgorithms, can be carried out on this processing unit. The SLAMalgorithm which results in the poses of the images and the coordinatesof a sparse point cloud can be performed on the processing unit of thecontroller. The resulting camera poses and the images can then betransmitted to a cloud server where the dense matching is performed.

The results after processing can be

-   -   a point cloud, particularly a colored point cloud;    -   a surface model, particularly a textured surface model;    -   an orthophoto, particularly of the ground or a façade; or    -   a set of salient points, e. g. corner points, automatically        derived from the point cloud.

For a completeness check the operator should receive already in thefield a preview model shortly, e. g. a few minutes, after the dataacquisition is finished. Based on this preview model the operator candecide whether he captured the measuring areas completely or whethersome parts are uncovered. In the latter case the operator can take someadditional measurements in the uncovered area to increase the level ofcompleteness.

The completeness check can be augmented with contrast analysis of theimages in order to detect areas that are missing and for which it isunlikely that they will be reconstructed (uniform surfaces). This wouldsave time of the user trying to reconstruct this kind of areas.

The decision can be done by the operator on the basis of a visualizationof the preview model, e. g. a 3D visualization. Alternatively, thevisualization can be 2D as a map view, to present the user with aquicker and easier-to-understand view of the model compared with 3Dmodels which are difficult to interpret and navigate for inexperiencedusers.

Particularly, the missing parts are high-lighted. Moreover, the systemcan guide the operator to the area of missing measurements.

The preview model should be available shortly after the data acquisitionis finished. In order to save time, e. g. for data transmission andprocessing, the preview model may have lower resolution than the finalmodel, e. g. a point resolution on the object of 10 cm.

This can be achieved by performing particularly the dense matching onimages with lower resolution, e. g. after a reduction the imageresolution from 1920×1080 pixels to 320×240 pixels.

In case of limited bandwidth, if the processing is carried out on acloud server the reduction of the image resolution is carried out beforethe data is transmitted. Moreover, the image data can be compressedbefore they are sent to the processing server. Therein, the compressionof image data can be carried out in a lossless way (i.e. a lossless datacompression can be applied, without reducing the quality/resolution ofthe images) or in a lossy way (i.e., depending on the needs andcircumstances of a particular situation, also a lossy data compressioncan be applied, including a reduction of the quality/resolution of theimages).

Moreover, the data can be reduced (areas of very low contrast do notneed to be transmitted) using sparse image representations.

By matching the computed orthophoto to the reference aerial image,additional location information can be derived. This can be used as theonly source of geo-location or can be coupled with GNSS or prism-basedterrestrial measurements in order to improve geo-location accuracy.

After the whole scene is captured, the SLAM algorithm is executed on thedataset. The structure from motion can use partial GNSS data, howeverthis is not required. After computing the external camera parameters,the orthophoto generation algorithm is executed. This algorithm works asfollows:

1) The points belonging to the ground surface are determined. This canbe done using IMU to get the vertical orientation vector and detectplanar surfaces that are perpendicular to this vector. Alternatively,movement directions from SLAM are used to estimate the vectors that areperpendicular to the ground.

2) After the ground surface determination, the points belonging to thissurface are triangulated, for example using 2D Delaunay triangulation.Alternatively, a set of planes is fitted to the ground surface.Alternatively, the ground surface points are filtered before fittingplanes or triangulating the surface.

3) Texture generation for the ground surface (now defined as a set ofplanes or triangles) is performed. Depending on the qualityrequirements, the texture can be obtained by colouring the surfaces fromthe point cloud. Alternatively this can be done by back-projecting theimages using the determined camera projection matrices. In the mostdemanding case, where the highest texture quality is required, theAnalysis by Synthesis approach is used. This works by back-projectingall captured images to the ground surface, averaging the colourcontributions on the ground surface, then projecting the parts of theground surface back to the cameras and computing the difference imagesbetween the captured images and projected ground surfaces. Then thedifference images are back-projected to the ground surfaces, the coloursare accumulated, to compute new improved textures of the groundsurfaces. The process is repeated then in a loop. This allowsreconstructing a high quality texture even from mages with highperspective distortions, where the ground surface is extremelyforeshortened.

4) The textured ground surface is then projected onto a ground plane inorder to obtain the orthophoto that can be matched to the geo-referencedorthophoto created from aerial images.

5) The camera positions are known with respect to the generatedorthophoto in a local coordinate system. The generated ground image isthen registered with the geo-referenced aerial image using purely imagebased methods. The camera positions then can be computed in thegeographic coordinate system, and the point cloud can be thentransformed to the geographic coordinate system.

In some applications interactive geolocation without GNSS signal may berequired. The following method then can be used:

1) During the data capture, only local texture is computed from a verylimited number of frames. This requires running the SLAM on a possiblysmall subset of cameras in order to discover their relations to eachother.

2) The local texture is then registered to the aerial image using imagebased methods (for example, feature matching)

3) The position of the camera and a model scale then can be determinedfrom a few local cameras and the registration of a local ground textureto the aerial image.

This approach can be used for an interactive visualization of the cameraposition on the map on the local device's screen. The local SLAMcomputations may be joined and optimized for a good fitting together inorder to find relations between different sub-reconstructions with SLAM.

In the case of the GNSS available, the method can be used to create ageo-referenced orthomap for use in other applications without the needof renting a plane or a UAV. FIG. 16a shows an example orthophoto 160generated from the images acquired at the ground level. As it can beseen, those areas 162 not captured from the ground level, such asbuilding roofs or tree tops, are not present. FIG. 16b presents anexample reference orthophoto 165 from a georeferenced aerial image.Those two images have a different scale and orientation. By usingmatching of scale and rotation of invariant features the generated imagecan be transformed to match the scale and rotation of the referenceaerial image (the borders of the orthophoto 160 from FIG. 16a arerepresented by the outline 166). This transformation is then used tobring the point cloud and positions of the measurement instrument to thecoordinate system of the reference image 165.

Pole-Based Scale Derivation

The camera module can also be used without being combined with a GNSSsystem or a total station, in particular in applications where noabsolute geo-referencing is needed.

This is illustrated in FIG. 17.

Disadvantageously, a 3D reconstruction without GNSS or total stationmight have an arbitrary scale. A state-of-the-art solution of this taskis to use reference points with known 3D coordinates. These points aremeasured manually or automatically on the images. Their 3D coordinatesare determined using additional measurement unit (e. g. with a totalstation). Alternatively, a scale bar could be used which isautomatically identified in the image data.

Alternatively, a method with the following steps could be used todetermine the scale:

1) The height of the camera module 30 which is mounted on a pole 10 ismeasured.

2) A user 2 starts recording images with a pole 10 standing on theground.

3) A user moves along the object and records images.

4) After the data processing, a dense point cloud of the object inarbitrary scale is reconstructed. A height of the camera module abovethe ground is determined in the reconstructed point cloud for the firstimage.

5) A scale is computed by comparison of heights determined on the steps1 and 4.

To improve an accuracy of scale computation, a user 2 could put the pole10 onto the ground more than one time during the survey. The user couldpress a button on the pole to indicate the moment when the pole 10 isstanding on the ground. Alternatively, this moment might be determinedautomatically based on comparison of sequentially recorded images or IMUinformation.

The whole reconstruction is done in an arbitrary oriented coordinatesystem. However, the user is able to define a local coordinate system bythe following procedure:

1) The user 2 places the pole 10 on the ground and orients the polevertically using an integrated bubble. After the user presses a buttonon a device which defines this point 5 on the ground as an origin of thecoordinate system XYZ, by default the first point 5 of the whole surveyis considered as an origin. The vertical axis Z of the coordinate systemis defined by a current orientation of the pole 10 relative to the plumbline 9.

2) For the definition of the horizontal orientation of the coordinatesystem a user 2 places the pole 10 in arbitrary orientation on a secondpoint 5′ onto the ground. A vertical orientation of the pole 10 is notimportant in this case. A vector from the origin of the coordinatesystem to this second point defines the horizontal orientation, e. g.the x-axis of the coordinate system.

The user could place the pole 10 on significant and marked points to beable to set the same coordinate system XYZ in the future (e. g. later inanother survey).

The height computation of the camera module 30 above the ground could bedone manually or automatically in the reconstructed point cloud. In thefirst case, a user selects points which belong to the ground. In thesecond case a height is extracted based on assumption that the pole 10is oriented vertically.

Alternatively, the height computation could also be performed in themeshed surface model.

Loop Closing

Optionally, the GNSS data in combination with a panorama camera can beused to extend the standard SLAM approach in a new way. GNSS data can beused to select two frames recorded at roughly the same position but atdifferent times. This happens when the user crosses his own path ortakes the same path multiple times. Because a GNSS pole is heldvertically under normal recording conditions, the main differencebetween the two frames is likely a horizontal rotation, i.e. a change inazimuth angle. Such a single rotation can be determined and compensatedfor efficiently using the raw images or detected feature points. Afteror during compensation of the azimuth change, optionally a compensationof the small change in the other two orientation angles can be performedin a similar way. Once the two frames are roughly aligned, a multitudeof additional matches between the images can be found between the twoframes by using a tracking algorithm such as KLT. The additional matchesimprove the connection between the two frames that are distant and timebut not in position, i.e. loop closing. This stabilizes the algorithmand enhances the accuracy by further reducing drifts (also in rotation)for large datasets.

Loop-closing—according to the invention—is based on capturing a firstseries of images of the surrounding with the at least one camera, thefirst series comprising an amount of images captured with differentposes of the cameras, the poses representing respective positions andorientations of the cameras. Furthermore, an initial set of image pointsis identified based on the first series of images, the initial imagepoints representing reference points 62 of an initial reference pointfield 65 (FIG. 14), wherein each reference point appears in at least twoimages of the series of images, and the poses for the images aredetermined based on resection using the initial image points.

According to the invention, a second series of images of the surroundingis captured with the at least one camera, reference points of thereference point field 65 appearing in at least one of the images of thesecond series of images are identified, a further set of image points isidentified in the images of the second series of images corresponding tothe identified reference points 62 of the second series of images, andthe poses for the images of the second series of images are determinedbased on resection using the initial set and the further set of imagepoints.

Wheel on the Bottom of the Pole

The FIGS. 18 and 19 a-d show two exemplary embodiments of a wheeledsurveying system according to the invention that facilitate thesurveying process for the user.

The wheeled surveying system 180 depicted in FIG. 18 comprises a pole 10comprising the features of the pole of FIG. 1. Additionally, the pole 10comprises wheels 181 that allow the user 2 to move the pole 10 along apath through the surrounding without having to carry the pole 10. Tofacilitate the pushing (or pulling) of the pole 10, it is equipped witha handle 182 (or two handles).

Preferably, two wheels 181 are attached to the pole 10 in such a waythat the bottom end of the pole touches the ground if the pole isvertical (i.e. in a 90° angle relative to the ground). Even morepreferably, the pole 10 keeps this upright position autonomously.

For moving the pole 10 it needs to be tilted (as shown in FIG. 18).

Alternatively, the bottom end of the pole 10 can be extendable in orderto touch the ground.

Optionally, the wheels 181 can be actuated by a motor, particularly anelectric motor.

This motor e. g. can either be controlled by the user 2 by a controlunit on the handle 182 or can act as a support drive for the pushing (orpulling) movements of the user 2.

The wheeled surveying system 190 depicted in FIGS. 19a-d comprises atwo-wheeled, self-balancing motorized vehicle 195. This kind of vehicleis also widely known as “Segway personal transporter”. The surveyingsystem 190 comprises a pole 10 which is mounted on the vehicle 195. Onthe pole 10, as already described with respect to FIG. 1, a GNSS antenna15 and a camera module 30 are provided. The surveying system 190 alsocomprises a control and evaluation unit 12.

As known from similar vehicles, a motion of the vehicle 195 can becontrolled by the user 2 as illustrated in FIGS. 19c and 19d : Thedepicted vehicle 195 is designed in such a way that the user 2 controlsa forward and backward movement of the vehicle 195 by leaning thevehicle relative to a combined centre of gravity of user 2 and vehicle195.

Optionally, the vehicle 195 can comprise measuring point marking means192 that mark a spot on the ground as a present measuring point 5.Particularly, this spot can be an imaginary extension of the pole 10.The marking can be a laser spot or pattern which lasts only for the stayat the respective position and is intended as information for the user 2of the surveying system 190 driving the vehicle 195. In particular, thisenables the user 2 to measure exactly at predefined marked positions.Alternatively, the marking can be more durable, for instance a sprayedpaint marking or a dropped flag. This allows repeating the measurementsat a later point of time at the same positions.

Optionally, an IMU of the vehicle 195 can be used for determiningorientations of the cameras of the camera module 30—either alternativelyor additionally to an IMU of the camera module 30.

As shown in FIG. 20, the pole 10 can be equipped with a rain and/or sunprotection, e. g. an umbrella being mounted below or around the GNSSantenna 15. This does not only facilitate the working conditions for theuser 2 but also protects the camera module 30 and other features of therespective surveying system 1,180,190 and improves the image quality incase of precipitation.

Although the invention is illustrated above, partly with reference tosome preferred embodiments, it must be understood that numerousmodifications and combinations of different features of the embodimentscan be made. All of these modifications lie within the scope of theappended claims.

What is claimed is:
 1. A surveying subsystem comprising a camera moduleand a control and evaluation unit, wherein the surveying subsystem isconfigured to be used as part of a surveying system adapted to determinepositions of a position measuring resource, particularly wherein theposition measuring resource is a GNSS-antenna or a retro-reflector, thecamera module comprising at least one camera for capturing images andparticularly being designed to be attached to the position measuringresource, the control and evaluation unit having stored a program withprogram code so as to control and execute a functionality in which—whenmoving along a path through a surrounding a series of images of thesurrounding is captured with the at least one camera, the seriescomprising an amount of images captured with different poses of thecamera, the poses representing respective positions and orientations ofthe camera; a SLAM-evaluation with a defined algorithm using the seriesof images is performed, wherein a plurality of respectivelycorresponding image points are identified in each of several sub-groupsof images of the series of images with use of a feature detectionprocess for detecting corresponding features, particularly by means offeature tracking, especially a KLT-algorithm (Kanade-Lucas-Tomasi),and/or by means of feature matching, especially a SIFT-, SURF-, BRISK-or BRIEF-algorithm, and, based on resection and forward intersectionusing the plurality of respectively corresponding image points, areference point field is built up comprising a plurality of referencepoints of the surrounding, wherein coordinates of the reference pointsare derived, and the poses for the images are determined, wherein thecontrol and evaluation unit comprises several diversified units, whereinat least a first unit, particularly an FPGA or a GPU (i.e. asabbreviation for graphical processing unit), carries out at least partof the feature detection process for identifying the plurality ofrespectively corresponding image points, and wherein at least another,second unit, particularly a CPU, carries out determining of the poses.2. The subsystem according to claim 1, wherein the first and secondunits are built together as one SoC (i.e. as abbreviation for system ona chip).
 3. The subsystem according to claim 1, wherein the first andthe second unit are disposed within the camera module.
 4. The subsystemaccording to claim 1, wherein the first unit is disposed within thecamera module and the second unit is a laptop, a tablet-PC, asmartphone, a processing unit of a handheld controller/datalogger of thesurveying system, a processing unit of a GNSS-module of the surveyingsystem, a processing unit of a surveying station, particularly totalstation, of the surveying system or a surveying site server,particularly being installed in a car.
 5. The subsystem according toclaim 1, wherein the camera module is designed to be attached to ahand-carried surveying pole as part of a surveying system that isadapted to determine positions of a position measuring resource, and theposition measuring resource also being mountable on the surveying pole.6. A surveying subsystem comprising a camera module and a control andevaluation unit, wherein the surveying subsystem is configured to beused as part of a surveying system adapted to determine positions of aposition measuring resource, particularly wherein the position measuringresource is a GNSS-antenna or a retro-reflector, the camera modulecomprising at least one camera for capturing images, wherein the cameramodule is built as one single integrated physical unit being designed tobe attached to the position measuring resource, and the control andevaluation unit having stored a program with program code so as tocontrol and execute a functionality in which—when moving along a paththrough a surrounding a series of images of the surrounding is capturedwith the at least one camera, the series comprising an amount of imagescaptured with different poses of the camera, the poses representingrespective positions and orientations of the camera, a SLAM-evaluationwith a defined algorithm using the series of images is performed,wherein a plurality of respectively corresponding image points areidentified in each of several sub-groups of images of the series ofimages and, based on resection and forward intersection using theplurality of respectively corresponding image points, a reference pointfield is built up comprising a plurality of reference points of thesurrounding, wherein coordinates of the reference points are derived,and the poses for the images are determined, a spatial representation,particularly a point cloud, comprising 3d-information about thesurrounding is computed by forward intersection using the series ofimages and the determined poses, particularly by using a dense matchingalgorithm, wherein the control and evaluation unit comprises severaldiversified units, wherein at least one unit, being integrated in thecamera module, carries out at least part of the SLAM-evaluation and atleast one other unit, being disposed externally to the camera module,carries out at least parts of computing of the spatial representation.7. The subsystem according to claim 6, wherein the at least one otherunit is a cloud server.
 8. The subsystem according to claim 1, whereinthe at least one other unit is a laptop, a tablet-PC, a smartphone, aprocessing unit of a handheld controller/datalogger of the surveyingsystem, a processing unit of a GNSS-module of the surveying system or asurveying site server, particularly being installed in a car.
 9. Thesubsystem according to claim 1, wherein several other units, beingdisposed externally to the camera module, carry out at least parts ofcomputing of the spatial representation in a decentralised manner, theseveral other units particularly being several smartphones.
 10. Thesubsystem according to claim 1, wherein at least one unit carrying out apart of identifying the plurality of respectively corresponding imagepoints and/or determining the poses is designed so as to be carriable ina backpack of an operator.
 11. The subsystem according to claim 1,wherein at least one unit carrying out a part of identifying theplurality of respectively corresponding image points and/or determiningthe poses is a laptop, a tablet-PC, a smartphone, a processing unit of ahandheld controller/datalogger of the surveying system, a processingunit of a GNSS-module of the surveying system or a surveying siteserver, particularly being installed in a car.
 12. The subsystemaccording to claim 1, wherein the camera module is designed to beattached to a hand-carried surveying pole as part of a surveying systemthat is adapted to determine positions of a position measuring resource,and the position measuring resource also being mountable on thesurveying pole.